Svc::DpCatalog Component

1 Introduction

DpCatalog is an active F´ component that manages the downlink of generate data products. The data products were previously generated by components into a project-specific set of directories. Upon command, DpCatalog will build a catalog of existing data products and start downloading them. They are downloaded based on a priority stored in the file and by generation time. DpCatalog also has commands to reprioritize and delete data products.

NOTE: This release has an early prototype that will do the following:

Reads the data product files from the specified directory.
Puts them in priority sorted list.
Sends requests to Svc/FileDownlink to downlink each file in the list.
Stop transmissions in progress
Allow insertion of new products after the catalog is built
Clear the catalog via command

It does not:

Send data product files in pieces. It only sends whole files, so if the downlink is interrupted, the whole file will have to be retransmitted.

Features and more extended unit testing will be added over time. Use at your own risk!

2 Requirements

Requirement	Description	Rationale	Verification Method
SVC-DPCAT-001	`DpCatalog` shall read a set of directories and build a catalog of data products.	`DpCatalog` needs to know at least one directory where data products reside	Test
SVC-DPCAT-002	`DpCatalog` shall sort data products first based on the internally recorded priority where the highest priority is represented by the lowest number.	`DpCatalog` needs to downlink highest priority items first	Test
SVC-DPCAT-003	`DpCatalog` shall sort data products second based on the internally recorded time with oldest products as higher priority.	`DpCatalog` needs to downlink oldest items first	Test
SVC-DPCAT-004	`DpCatalog` shall sort data products third based on the internally recorded product ID with the lowest as higher priority.	`DpCatalog` needs to resolve case where priority and time match	Test
SVC-DPCAT-005	`DpCatalog` shall update the data product metadata once download is complete	`DpCatalog` needs to track completion status to avoid duplicate downloads	Test
SVC-DPCAT-006	`DpCatalog` shall implement a command and port to build the catalog.	`DpCatalog` should consume the resources to build the catalog only when needed	Test
SVC-DPCAT-007	`DpCatalog` shall implement a command and port to start downloading the catalog.	`DpCatalog` downloads should be timed for when communications are ready, and not consume comm window time building	Test
SVC-DPCAT-008	`DpCatalog` shall have filters for the download based on container ID, priority, and data size limit.	`DpCatalog` downloads should be tunable for available bandwidth and container metadata	Test
SVC-DPCAT-009	`DpCatalog` shall implement a way to insert newly-generated data products into the catalog after the catalog is built	`DpCatalog` should notice new products and not require a rebuild of the catalog	Test
SVC-DPCAT-010	`DpCatalog` shall implement commands to modify the priority of existing data products	Operators made change the priority to lower or raise priorities due to troubleshooting, etc	Test
SVC-DPCAT-011	`DpCatalog` shall implement commands to delete data products	Ground tools can use DpManager commands to automatically delete DPs	Test
SVC-DPCAT-012	`DpCatalog` shall implement a catalog data product that is a listing of existing data products and their metadata	Allow operators to know what products exist	Test
SVC-DPCAT-013	`DpCatalog` shall have filters for the catalog data product for priority, container ID, and time range	Allow operators to know what products exist	Test

3 Design

3.1 Assumptions

The design of DpCatalog assumes the following:

A file system exists to store the data product files.
The contents of the data product files match the data product specification.
The file downlink will acknowledge completion of each file

3.2 Ports

3.2.1 Role Ports

These ports will be automatically connected in the topology to F Prime services.

Name	Role
cmdDisp	Receives commands
CmdReg	Registers commands
CmdStatus	Returns command status
Log	Outputs events for ground
LogText	Outputs events for console
Time	Gets time for time tags
Tlm	Outputs telemetry

3.2.2 Component-Specific Ports

Name	Type	Kind	Purpose
pingIn	Svc.Ping	async input	Ping from Health
pingOut	Svc.Ping	output	Ping response to Health
fileOut	Svc.SendFileRequest	output	Send next file to downlink
fileDone	Svc.SendFileComplete	async input	Last requested file is complete
addToCat	Svc.DpWritten	async input	Insert a newly written data product into the catalog

3.3 Constants

DpCatalog can be statically configured with the following constants:

Constant	Purpose
DP_MAX_DIRECTORIES	Maximum directories that can be provided for DPs
DP_MAX_FILES	Maximum number of files that can be tracked across directories

These constants are located in DpCatalogCfg.hpp in the config directory.

3.4 Configuration

During initialization, the configuration function takes a set of parameters:

        void configure(
            const Fw::ExternalArray<Fw::FileNameString>& directories,
            Fw::FileNameString& stateFile,
            FwEnumStoreType memId,
            Fw::MemAllocator& allocator
        );

Parameter	Purpose
`directories`	An array of strings up to `DP_MAX_DIRECTORIES` that are directory names where DPs are written
`stateFile`	The location of the file tracking product downlink state
`memId`	The id of the RAM memory segment used to store catalog state. Not needed for heap allocation.
`allocator`	Memory allocator for RAM memory storage

3.5 Commands

Command	Arguments	Description
`BUILD_CATALOG`	none	Builds the in-RAM catalog by scanning the directories provided during initialization. Downlink state file will be read in to set downlink state for products
`START_XMIT_CATALOG`		Start transmitting the catalog to the ground in priority order
	wait	Wait for the transmission to complete before sending command completion status. Used when a sequence wishes to wait for completion before issuing subsequent commands.
`STOP_XMIT_CATALOG`	none	Stop existing catalog transmission. The command completes immediately and no further files are started; the file in flight completes normally and is recorded. Its bytes are counted in neither `CatalogXmitStopped` (emitted before that completion) nor a later `CatalogXmitCompleted` (a `START_XMIT_CATALOG` issued after the completion starts a fresh tally); reconstruct them from `ProductComplete`. A `START_XMIT_CATALOG` (or a runtime `addToCat` with `remainActive`) issued before that completion resumes the transmission without re-sending the file: no new send is started until its completion arrives, which then continues the catalog walk, and the session's byte tally reported by `CatalogXmitCompleted` is kept. As with any send, the resumed transmission stays in progress until FileDownlink delivers that completion; if it never arrives, `STOP_XMIT_CATALOG` (then `BUILD_CATALOG`) or `CLEAR_CATALOG` is the recovery, as for a stalled send started normally. A `BUILD_CATALOG` or `CLEAR_CATALOG` issued before that completion abandons the file: its late completion is reported as `StaleFileDone` and it is re-sent on the next `START_XMIT_CATALOG`.
`CLEAR_CATALOG`	none	Clears existing RAM catalog and resets downlink state, reporting the pending products and bytes dropped with `CatalogCleared`. A transmission in progress is aborted (a waited `START_XMIT_CATALOG` is answered with `EXECUTION_ERROR`). Should be followed by `BUILD_CATALOG`. Used for recovery if state file gets corrupted or out of sync with file system contents.

Sequence of Commands

When the software is first started, the catalog data structure is empty. The catalog must be built before starting downlink. The BUILD_CATALOG command was implemented separately from the START_XMIT_CATALOG command to start downlinking so the software could build the catalog prior to a communication session and execute the downlink during communication. Downlink can be halted by issuing the STOP_XMIT_COMMAND in the middle of the downlink. If for some reason the state in the state file, the contents of the tree and the data products get out of sync, a CLEAR_CATALOG command can be issued. Then the BUILD_CATALOG command can be invoked to rebuild the state based on the existing set of data product files only. This will caused downlinked state to be lost, so data products not deleted after downlink would be readded to the pending list of downlinks.

3.6 Algorithms

3.6.1 Data Product Sorting

Overall priority is determined by comparing two records' priority, timestamp, and id in that order with lower numerical values holding higher priority (see DpCatalog::DpStateEntry::compareEntries). The following logic is implemented:

Data Product Priority - Data products are generated with a priority, where the lower the number the higher the priority.
Data Product Generation Time - If priorities are the same, the older data is prioritized over the newer.
Data Product ID - If priorities and time are the same (highly unlikely), then lower IDs are prioritized first.

3.6.2 Reading Files

The configure() function is provided an array of directories where data product files are generated by Svc/DpWriter. When the BUILD_CATALOG command is executed, the headers of the data product files are read and the metadata in their headers is processed and stored as a data structure for sorting. The file name is not stored to conserve memory.

3.6.3 Sorting Algorithm

The data products are sorted using a Fw::RedBlackTreeSet (a self-balancing binary search tree). Entries are ordered by the priority comparison described above, so the highest priority entry is always first in the set.

3.6.4 Tree Traversal for Downlink

When data products are downlinked, entries are retrieved in priority order by calling begin() on the set, which always yields the highest priority entry. As each file completes downlink, its entry is removed from the set. Entries inserted while a downlink is in progress (e.g. via addToCat) are placed in priority order and picked up by subsequent begin() calls.

3.6.5 State File

When a data product is downlinked, it is marked in the node as completed, but the state is also written to a file so that downlinked state is preserved across restarts of the software. When the catalog is built, the state file is first read into a data structure in memory. Entries whose state-file record is TRANSMITTED are skipped during catalog build, emit a DpFileSkipped event, and are not counted as pending.

3.6.6 FileDone Handling

Every sendFile call returns a SendFileResponse whose context FileDownlink assigns to that send and echoes back in fileDone. DpCatalog keeps the context of the send in flight and applies a fileDone only while a send is in flight and the context matches. Anything else is a late callback from a send abandoned by BUILD_CATALOG or CLEAR_CATALOG (STOP_XMIT_CATALOG alone keeps the send in flight, so its completion is still recorded, and a START_XMIT_CATALOG issued before it arrives starts no new send until then, so the file is not sent twice): it is reported with StaleFileDone (WARNING_LO, id 50, an expected consequence of an operator command rather than a fault) and the transmit in flight, if any, is left untouched. CLEAR_CATALOG issued mid-transmit closes the transmit session itself, as STOP_XMIT_CATALOG does: a waited START_XMIT_CATALOG is answered with EXECUTION_ERROR at that point, so a later BUILD_CATALOG or START_XMIT_CATALOG is not refused as in progress and the late callback is only reported. This replaces the FW_ASSERT that made a late fileDone FATAL (#5777). The discrimination requires the provider to assign a distinct context to every send, as Svc/FileDownlink does; a provider that echoes a constant context (for example Svc/Ccsds/CfdpManager, which returns the port number) cannot distinguish a late callback from the current send and is not supported on this port. Note: FileComplete must not be shared with other SendFile clients on the same FileDownlink; foreign completions would be reported as StaleFileDone.

4 Unit Testing

Unit tests are located in Svc/DpCatalog/test/ut.