Sarracenia

https://metpx.github.io/sarracenia
Admin
Sarracenia is a real-time pub/sub data sharing management toolkit for command line, Python, and C...
Tokens:614,737
Snippets:6,393
Trust Score:9.7
Update:3 months ago
Show doc for...
Tokens:
Raw
### List Example Configurations with sr3

Source: https://metpx.github.io/sarracenia/Explanation/CommandLineGuide.html

Use 'sr3 list examples' to view included configuration templates that can be used as starting points for new configurations.

```bash
$ sr3 list examples
  Sample Configurations: (from: /home/peter/Sarracenia/development/sarra/examples )
  cpump/cno_trouble_f00.inc        poll/aws-nexrad.conf             poll/pollingest.conf
  poll/pollnoaa.conf               poll/pollsoapshc.conf            poll/pollusgs.conf
  poll/pulse.conf                  post/WMO_mesh_post.conf          sarra/wmo_mesh.conf
  sender/ec2collab.conf            sender/pitcher_push.conf         shovel/no_trouble_f00.inc
  subscribe/WMO_Sketch_2mqtt.conf  subscribe/WMO_Sketch_2v3.conf    subscribe/WMO_mesh_CMC.conf
  subscribe/WMO_mesh_Peer.conf     subscribe/aws-nexrad.conf        subscribe/dd_2mqtt.conf
  subscribe/dd_all.conf            subscribe/dd_amis.conf           subscribe/dd_aqhi.conf
  subscribe/dd_cacn_bulletins.conf subscribe/dd_citypage.conf       subscribe/dd_cmml.conf
  subscribe/dd_gdps.conf           subscribe/dd_ping.conf           subscribe/dd_radar.conf
  subscribe/dd_rdps.conf           subscribe/dd_swob.conf           subscribe/ddc_cap-xml.conf
  subscribe/ddc_normal.conf        subscribe/downloademail.conf     subscribe/ec_ninjo-a.conf
  subscribe/hpfx_amis.conf         subscribe/local_sub.conf         subscribe/pitcher_pull.conf
  subscribe/sci2ec.conf            subscribe/subnoaa.conf           subscribe/subsoapshc.conf
  subscribe/subusgs.conf           watch/master.conf                watch/pitcher_client.conf
  watch/pitcher_server.conf        watch/sci2ec.conf


```

--------------------------------

### Start Message Publishers

Source: https://metpx.github.io/sarracenia/_modules/sarracenia/flowcb/post/message.html

Initiates the setup process for all message publishers. This method should be called when the service starts to ensure publishers are ready to send messages.

```python
    def on_start(self):
        for p in self.posters:
            if hasattr(p,'putSetup'):
                p.putSetup()

        logger.debug('starting')
```

--------------------------------

### Example Subscriber Configuration for Troubleshooting

Source: https://metpx.github.io/sarracenia/_sources/How2Guides/subscriber.rst.txt

This example demonstrates how to create a sample subscriber configuration to identify slow downloads by examining lag in the download log. It's a starting point for diagnosing performance issues.

```shell
fractal% sr3 list ie

 Sample Configurations: (from: /home/peter/Sarracenia/sr3/sarracenia/examples )
 cpump/cno_trouble_f00.inc        flow/amserver.conf               flow/poll.inc                    flow/post.inc                    flow/report.inc                  flow/sarra.inc                   
 flow/sender.inc                  flow/shovel.inc                  flow/subscribe.inc               flow/watch.inc                   flow/winnow.inc                  poll/airnow.conf                 
 poll/aws-nexrad.conf             poll/mail.conf                   poll/nasa-mls-nrt.conf           poll/noaa.conf                   poll/soapshc.conf                poll/usgs.conf                   
 post/WMO_mesh_post.conf          sarra/wmo_mesh.conf              sender/am_send.conf              sender/ec2collab.conf            sender/pitcher_push.conf         shovel/no_trouble_f00.inc        
 subscribe/aws-nexrad.conf        subscribe/dd_2mqtt.conf          subscribe/dd_all.conf            subscribe/dd_amis.conf           subscribe/dd_aqhi.conf           subscribe/dd_cacn_bulletins.conf 
 subscribe/dd_citypage.conf       subscribe/dd_cmml.conf           subscribe/dd_gdps.conf           subscribe/dd_radar.conf          subscribe/dd_rdps.conf           subscribe/dd_swob.conf           
 subscribe/ddc_cap-xml.conf       subscribe/ddc_normal.conf        subscribe/downloademail.conf     subscribe/ec_ninjo-a.conf        subscribe/hpfxWIS2DownloadAll.conf subscribe/hpfx_amis.conf         
 subscribe/hpfx_citypage.conf     subscribe/local_sub.conf         subscribe/ping.conf              subscribe/pitcher_pull.conf      subscribe/sci2ec.conf            subscribe/subnoaa.conf           
 subscribe/subsoapshc.conf        subscribe/subusgs.conf           sender/am_send.conf              sender/ec2collab.conf            sender/pitcher_push.conf         watch/master.conf                
 watch/pitcher_client.conf        watch/pitcher_server.conf        watch/sci2ec.conf                
fractal% 

pick one ane add it local configuration::
```

--------------------------------

### List Available Sarracenia Configuration Examples

Source: https://metpx.github.io/sarracenia/_sources/Tutorials/1_CLI_introduction.ipynb.txt

Lists all available sample configuration files for Sarracenia, organized by type (e.g., flow, poll, subscribe). These examples can be used as a starting point for defining custom data flows.

```bash
sr3
```

--------------------------------

### Command Line Execution Style Guide

Source: https://metpx.github.io/sarracenia/_sources/Contribution/Documentation.rst.txt

Demonstrates the standard format for writing command-line execution examples in documentation. Includes an initial comment, commands prefixed with '$', and indented relevant output.

```bash
An initial comment describing the following steps or processes::

    $ command 1
      relevant output
    $command 2
      .
      .
      relevant output
      newline relevant output
```

--------------------------------

### Start All Sarracenia Components

Source: https://metpx.github.io/sarracenia/_sources/Explanation/CommandLineGuide.rst.txt

Launches all configured components by executing the 'start' command. This includes gathering global state before starting.

```bash
$ sr3 start
    gathering global state: procs, configs, state files, logs, analysis - Done. 
    starting...Done
```

--------------------------------

### Python Logging Setup

Source: https://metpx.github.io/sarracenia/Tutorials/2_CLI_with_flowcb_demo.html

Demonstrates the standard Pythonic way to set up and use the built-in logging module. Import the module and get a logger instance for the current file.

```python
import logging

logger = logging.getLogger(__name__)

logger.debug('got here')
```

--------------------------------

### Install All Extras via Pip

Source: https://metpx.github.io/sarracenia/Tutorials/Install.html

Install metpx-sr3 with all available extras using the 'all' shortcut with pip.

```bash
$ pip install metpx-sr3[all]
```

--------------------------------

### Start All Active SR3 Configurations

Source: https://metpx.github.io/sarracenia/Tutorials/1_CLI_introduction.html

Use the `!sr3 start` command to start all active configurations across all SR3 components. This is a convenient way to bring up the entire SR3 system.

```bash
!sr3 start

```

--------------------------------

### Sarracenia Flow Configuration Example

Source: https://metpx.github.io/sarracenia/_sources/Tutorials/3_api_flow_demo.ipynb.txt

Illustrates the initialization of Sarracenia's configuration with various options related to message processing, retry mechanisms, and logging. This setup is typically part of launching a Sarracenia flow.

```python
cfg = sarracenia.config.Config()
cfg.add_option('messageCountMax', 1000000, 'max messages to process')
cfg.add_option('retry_queues', ['work_retry_00'], 'retry queues')
cfg.add_option('retry_driver', 'disk', 'retry driver type')
cfg.add_option('retry_refilter', False, 'refilter retried messages')
cfg.add_option('MemoryMax', 0, 'max memory to use')
cfg.add_option('MemoryBaseLineFile', 100, 'baseline memory file size')
cfg.add_option('MemoryMultiplier', 3, 'memory multiplier')
cfg.add_option('logEvents', {'after_post', 'after_work', 'on_housekeeping', 'after_accept'}, 'events to log')
cfg.add_option('logMessageDump', False, 'log message dumps')

# masks are a compiled form of accept/reject directives.
# a relPath is compared to the regex in the mask.
# If the regex matches, and accept is true, then the message is accepted for further processing.
# If the regex matches, but accept is False, then processing of the message is stopped (the message is rejected.)
# masks are a tuple. the meaning can be looked up in the sr3(1) man page.
#
# *  pattern_string,      the input regular expression string, to be compiled by re routines.
# *  directory,           where to put the files downloaded (root of the tree, when mirroring)
# *  fn,                  transformation of filename to do. None is the 99% use case.
# *  regex,               compiled regex version of the pattern_string
# *  accept(True/False),  if pattern matches then accept message for further processing.
# *  mirror(True/False),  when downloading build a complete tree to mirror the source, or just dump in directory
# *  strip(True/False),   modify the relpath by stripping entries from the left.
# *  pstrip(True/False),  strip entries based on patterm
# *  flatten(char ... '/' means do not flatten.) )

# These settings are needed because they would ordinarily be set by the sarracenia.instance class which is
# normally used to launch flows. They allow setting up of run-time paths for retry_queues, and statefiles,
# to remember settings if need be between runs.
cfg.add_option('no', '/tmp/sarracenia_demo/no', 'path for no messages')
cfg.add_option('pid_filename', '/tmp/sarracenia_demo/pid', 'pid filename')

# Example of a mask
# cfg.masks = [
#     ('^.*$', '/', None, None, True, False, False, False, '/'),
# ]

```

--------------------------------

### Generate Windows Installer on Ubuntu

Source: https://metpx.github.io/sarracenia/Contribution/Release.html

Commands to set up the environment and generate a Windows installer on an Ubuntu system. Requires NSIS, pynsist, and wheel to be installed. Uses a specific Python version and redirects output to a log file.

```bash
# if not already installed:
sudo apt install nsis
pip3 install pynsist wheel

git checkout pre-release
./generate-win-installer.sh 3.10.11 2>&1 > log.txt
```

--------------------------------

### Example of flow_setup.sh execution

Source: https://metpx.github.io/sarracenia/Contribution/Development.html

Demonstrates the output of the flow_setup.sh script, including log cleaning messages.

```bash
blacklab% ./flow_setup.sh
cleaning logs, just in case
rm: cannot remove '/home/peter/.cache/sarra/log/*': No such file or directory
```

--------------------------------

### Sr3 CLI: Start Subscribe Configuration

Source: https://metpx.github.io/sarracenia/How2Guides/UPGRADING.html

In Sr3, the command-line interface uses a single entry point 'sr3'. This example shows the pattern for starting a subscription configuration.

```bash
sr3 start subscribe/config
```

--------------------------------

### Instance Startup and Configuration Loading

Source: https://metpx.github.io/sarracenia/_modules/sarracenia/instance.html

Handles instance startup, including setting up logging, permissions, and loading configuration. It prepares the environment by setting environment variables declared in the configuration and then initializes and runs the core instance.

```python
                if hasattr(cfg_preparse, 'permLog'):
                    os.chmod(logfilename, cfg_preparse.permLog)

                # FIXME: https://docs.python.org/3/library/contextlib.html portable redirection...
                if sys.platform != 'win32':
                    os.dup2(handler.stream.fileno(), 1)
                    os.dup2(handler.stream.fileno(), 2)

        else:
            logger.setLevel(logLevel)

        signal.signal(signal.SIGTERM, self.stop_signal)
        signal.signal(signal.SIGINT, self.stop_signal)

        if cfg_preparse.statehost:
            hostdir = cfg_preparse.hostdir
        else:
            hostdir = None

        if not os.path.isdir(os.path.dirname(pidfilename)):
            pathlib.Path(os.path.dirname(pidfilename)).mkdir(parents=True, exist_ok=True)

        with open(pidfilename, 'w') as pfn:
            pfn.write('%d' % os.getpid())

        cfg = sarracenia.config.one_config(component, config, cfg_preparse.action, isPost=False, hostDir=hostdir)

        cfg.novipFilename = pidfilename.replace(".pid", ".noVip")

        if not hasattr(cfg, 'env_declared'):
            sys.exit(0)

        for n in cfg.env_declared:
            os.environ[n] = cfg.env[n]
            os.putenv(n, cfg.env[n])

        self.o = cfg
        self.running_instance = Flow.factory(cfg)

        self.running_instance.run()

        if os.path.isfile(pidfilename):
            os.unlink(pidfilename)
        sys.exit(0)
```

--------------------------------

### Install Sarrac C Client using PPA

Source: https://metpx.github.io/sarracenia/_sources/Tutorials/Install.rst.txt

Install the Sarrac C client on Ubuntu using a PPA. This method ensures you get prebuilt binaries and necessary dependencies.

```bash
$ sudo add-apt-repository ppa:ssc-hpc-chp-spc/metpx
$ sudo apt-get update
$ sudo apt-get install metpx-sr3c
```

--------------------------------

### Command Line Syntax Comparison v2 vs v3

Source: https://metpx.github.io/sarracenia/How2Guides/Plugins_v2ToSr3.html

Illustrates the difference in command-line entry points for starting components between Sarracenia v2 and v3.

```bash
v2:  sr_*component* start config

v3:  sr3 start *component*/config

```

--------------------------------

### Install and Configure RabbitMQ

Source: https://metpx.github.io/sarracenia/Tutorials/Setup_a_local_subscriber.html

Installs RabbitMQ server, lists users, adds a new user 'bob' with a password, and lists virtual hosts. This is the initial setup for the local broker.

```bash
$ sudo apt install rabbitmq-server
$ sudo rabbitmqctl list_users
  Listing users ...
  user    tags
  guest   [administrator]

$ sudo rabbitmqctl add_user 'bob'
  Adding user "bob" ...
  Password: robert

$ sudo rabbitmqctl list_vhosts
  Listing vhosts ...
  name
  /

```

--------------------------------

### Upload to PyPI using setup.py (Deprecated)

Source: https://metpx.github.io/sarracenia/_sources/Contribution/Release.rst.txt

This command was used on older systems to upload a new release to PyPI. It is now deprecated.

```bash
python3 setup.py bdist_wheel upload
```

--------------------------------

### Regex Group Compatibility: Sundew Groups Start from 0 (True)

Source: https://metpx.github.io/sarracenia/_sources/Reference/sr3_options.7.rst.txt

When 'sundew_compat_regex_first_match_is_zero' is True, Sundew group numbering (starting from 0) is used, differing from Python's convention where group 0 is the entire string and subsequent groups start from 1. This example shows the resulting directory structure.

```text
* sundew_compat_regex_first_match_is_zero: True
* input url:  https://hpfx.collab.science.gc.ca/20231127/WXO-DD/meteocode/que/cmml/TRANSMIT.FPCN71.11.27.1000Z.xml
* accept pattern: .*/WXO-DD/meteocode/(atl|ont|pnr|pyr|que)/.*/TRANSMIT\.FP([A-Z][A-Z]).*([0-2][0-9][0-6][0-9]Z).*
* directory setting: /tmp/meteocode/${2}/${0}/${1}
* resulting directory:  /tmp/meteocode/1000Z/que/CN
```

--------------------------------

### Start All Sarracenia Components

Source: https://metpx.github.io/sarracenia/How2Guides/Admin.html

Use this command to start all components with named configurations. Ensure configurations for sarra, subscribe, winnow, and log are set up.

```bash
sr3 start

```

--------------------------------

### AMQP Topic Structure Example

Source: https://metpx.github.io/sarracenia/Reference/sr_post.7.html

Illustrates the complete topic structure for a notification message, starting with the protocol version and file path.

```text
v03.a.b.c.d
```

--------------------------------

### Minimal Sarracenia Configuration Options

Source: https://metpx.github.io/sarracenia/_modules/sarracenia.html

Example of a minimal options dictionary for Sarracenia consumer setup. It includes broker credentials and topic prefixes.

```python
options = sarracenia.config.no_file_config()
options['broker'] = sarracenia.config.credentials.Credential(
          'amqps://anonymous:anonymous@hpfx.collab.science.gc.ca')
options['topicPrefix'] = [ 'v02', 'post' ]
options['subscriptions'] = sarracenia.config.subscription.Subscriptions( [ 
               sarracenia.config.subscription.Subscriptions( 
                  options, 
                  options['queueName'],
                  'q_anonymous_' + socket.getfqdn() + '_SomethingHelpfulToYou',
                  '#' 
               ) ] )
```

--------------------------------

### Start All Components

Source: https://metpx.github.io/sarracenia/Explanation/CommandLineGuide.html

The 'start' command launches all configured components of the sarracenia project. It first gathers global state including processes, configurations, state files, logs, and analysis before initiating the startup sequence.

```bash
$ sr3 start
  gathering global state: procs, configs, state files, logs, analysis - Done.
  starting...Done

```

--------------------------------

### Add a new configuration from an example

Source: https://metpx.github.io/sarracenia/_sources/Explanation/CommandLineGuide.rst.txt

The 'sr3 add' command copies a configuration template from the examples directory to the user's configuration directory. The output shows the source and destination paths.

```bash
$ sr3 add dd_all.conf
    add: 2021-01-24 18:04:57,018 [INFO] sarracenia.sr add copying: /usr/lib/python3/dist-packages/sarracenia/examples/subscribe/dd_all.conf to /home/peter/.config/sr3/subscribe/dd_all.conf 
```

--------------------------------

### Sr3 CLI: Multiple Configurations

Source: https://metpx.github.io/sarracenia/How2Guides/UPGRADING.html

Sr3 allows specifying multiple configurations in a single command. This example demonstrates starting poll, subscribe, and sender configurations simultaneously.

```bash
sr3 start poll/airnow subscribe/airnow sender/cmqb
```

--------------------------------

### Regex Group Compatibility: Python Groups Start from 1 (False)

Source: https://metpx.github.io/sarracenia/_sources/Reference/sr3_options.7.rst.txt

When 'sundew_compat_regex_first_match_is_zero' is False, the system conforms to Python's regular expression group numbering, where group 0 is the entire string and subsequent groups start from 1. This example shows the directory setting required to achieve the same result as the previous snippet.

```text
* sundew_compat_regex_first_match_is_zero: False
* directory setting: /tmp/meteocode/${3}/${1}/${2}
* to get the same result.
```

--------------------------------

### List Available Configurations with sr3

Source: https://metpx.github.io/sarracenia/How2Guides/subscriber.html

Use 'sr3 list' to see all available sample and user configurations. This command helps in discovering pre-defined setups.

```bash
$ sr3 list examples
```

--------------------------------

### Output of Flow Setup Script

Source: https://metpx.github.io/sarracenia/_sources/Contribution/Development.rst.txt

This is the typical output observed when running the flow_setup.sh script, indicating the cleaning of logs and the copying of various configuration and example files for the flow test environment.

```bash
blacklab% ./flow_setup.sh
cleaning logs, just in case
rm: cannot remove '/home/peter/.cache/sarra/log/*': No such file or directory
Adding flow test configurations...
2018-02-10 14:22:58,944 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpump/cno_trouble_f00.inc to /home/peter/.config/sarra/cpump/cno_trouble_f00.inc.
2018-02-10 09:22:59,204 [INFO] copying /home/peter/src/sarracenia/sarra/examples/shovel/no_trouble_f00.inc to /home/peter/.config/sarra/shovel/no_trouble_f00.inc
2018-02-10 14:22:59,206 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpost/veille_f34.conf to /home/peter/.config/sarra/cpost/veille_f34.conf.
2018-02-10 14:22:59,207 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpump/pelle_dd1_f04.conf to /home/peter/.config/sarra/cpump/pelle_dd1_f04.conf.
2018-02-10 14:22:59,208 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpump/pelle_dd2_f05.conf to /home/peter/.config/sarra/cpump/pelle_dd2_f05.conf.
2018-02-10 14:22:59,208 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpump/xvan_f14.conf to /home/peter/.config/sarra/cpump/xvan_f14.conf.
2018-02-10 14:22:59,209 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpump/xvan_f15.conf to /home/peter/.config/sarra/cpump/xvan_f15.conf.
2018-02-10 09:22:59,483 [INFO] copying /home/peter/src/sarracenia/sarra/examples/poll/f62.conf to /home/peter/.config/sarra/poll/f62.conf
2018-02-10 09:22:59,756 [INFO] copying /home/peter/src/sarracenia/sarra/examples/post/shim_f63.conf to /home/peter/.config/sarra/post/shim_f63.conf
2018-02-10 09:23:00,030 [INFO] copying /home/peter/src/sarracenia/sarra/examples/post/test2_f61.conf to /home/peter/.config/sarra/post/test2_f1.conf
2018-02-10 09:23:00,299 [INFO] copying /home/peter/src/sarracenia/sarra/examples/report/tsarra_f20.conf to /home/peter/.config/sarra/report/tsarra_f20.conf
2018-02-10 09:23:00,561 [INFO] copying /home/peter/src/sarracenia/sarra/examples/report/twinnow00_f10.conf to /home/peter/.config/sarra/report/twinnow00_f10.conf
```

--------------------------------

### On Start Entry Point

Source: https://metpx.github.io/sarracenia/_sources/How2Guides/FlowCallbacks.rst.txt

Called after connection establishment and instantiation, but before any notification message transfer.

```python
def on_start(self):
         After the connection is established with the broker and things are instantiated, but
         before any notification message transfer occurs.
```

--------------------------------

### Get detailed information for a specific configuration

Source: https://metpx.github.io/sarracenia/_sources/Explanation/CommandLineGuide.rst.txt

When 'sr3 status' indicates a configuration is partial, use 'sr3 dump' with 'grep' to retrieve detailed information, including the command line used to start it.

```bash
$ sr3 dump | grep sender/tsource2send_f50
    49308: name:sr3_sender.py cmdline:['/usr/bin/python3', '/usr/lib/python3/dist-packages/sarracenia/instance.py', '--no', '1', 'start', 'sender/tsource2send_f50']
    q_tsource.sr_sender.tsource2send_f50.58710892.12372870: ['sender/tsource2send_f50']
```

--------------------------------

### LongFlow plugin example for self-test

Source: https://metpx.github.io/sarracenia/Reference/flowcb.html

This plugin is for self-test purposes. It creates a header 'toolong' longer than 255 characters, which gets truncated. It helps test truncation handling and associated warning messages.

```python
class sarracenia.flowcb.accept.longflow.LongFlow(_options_)[source]:
    
    Bases: `FlowCB` 

    __init__(_options_)[source]
```

--------------------------------

### Initialize and Configure Sarracenia

Source: https://metpx.github.io/sarracenia/api-documentation.html

Demonstrates how to initialize an empty configuration and manually set various parameters including broker credentials, topic prefixes, component settings, and subscription details. This is useful for setting up a custom Sarracenia configuration programmatically.

```python
cfg = no_file_config()

cfg.broker = sarracenia.config.credentials.Credential('amqps://anonymous:anonymous@hpfx.collab.science.gc.ca')
cfg.topicPrefix = [ 'v02', 'post']
cfg.component = 'subscribe'
cfg.config = 'flow_demo'
cfg.action = 'start'
cfg.subscriptions= [ { 
    'broker': cfg.broker,
    'bindings': { 'exchange': 'xpublic', 'prefix': ['v02', 'post'], 'sub': [ '*.WXO-DD.observations.swob-ml.#' ] },
    'queue': {
        'name' : 'q_anonymous.subscriber_test2',
        'template':'q_${BROKER_USER}_${COMPONENT}_${CONFIG}_${HOSTNAME}_Demo123' ,
        'expire': 600,
        }
    } ]
cfg.download=True
cfg.batch=1
cfg.messageCountMax=5

# set the instance number for the flow class.
cfg.no=0


```

--------------------------------

### Install Sarracenia using pip (with Extras)

Source: https://metpx.github.io/sarracenia/_sources/Tutorials/Install.rst.txt

Command to install Sarracenia with additional features like AMQP, MQTT, VIP, FTP poll, and file type support. Use 'all' for all available extras.

```bash
pip install metpx-sr3[amqp,mqtt,vip,ftppoll,filetype]
```

--------------------------------

### Example Configuration for Standard Format Tree

Source: https://metpx.github.io/sarracenia/Reference/sr3_options.7.html

This configuration combines mirror, sourceFromExchange, and directory settings to ensure data arrives in the standard format tree. It's a common setup for structured data ingestion.

```plaintext
*mirror true*
*sourceFromExchange true*
*directory ${PBD}/${YYYYMMDD}/${SOURCE}*
```

--------------------------------

### Build and Install RPM Package

Source: https://metpx.github.io/sarracenia/Tutorials/Install.html

Clone the Sarracenia repository, navigate into the directory, build an RPM package using setuptools, and then install the generated RPM file.

```bash
$ git clone https://github.com/MetPX/sarracenia
$ cd sarracenia

$ python3 setup.py bdist_rpm
$ sudo rpm -i dist/*.noarch.rpm
```

--------------------------------

### Read and Write Data Chunk

Source: https://metpx.github.io/sarracenia/_modules/sarracenia/transfer.html

Resets transfer statistics such as total bytes transferred, start time, and last log time before initiating a read/write operation. This method is a setup for transferring data between source and destination.

```python
    def read_write(self, src, dst, length=0):
        logger.debug("sr_proto read_write")

        # reset speed

        rw_length = 0
        self.tbytes = 0.0
        self.tbegin = nowflt()
        self.lastLog = self.tbegin
```

--------------------------------

### Example Data Products from a Source

Source: https://metpx.github.io/sarracenia/How2Guides/subscriber.html

This snippet illustrates the organization of data products under a specific source directory on a Sarracenia data pump.

```html
Icon  Name                    Last modified      Size  Description
[TXT] about_dd_apropos.txt    2021-05-17 13:23   1.0K
[DIR] air_quality/            2020-12-10 14:47    -
[DIR] alerts/                 2022-07-13 12:00    -
[DIR] analysis/               2022-07-13 13:17    -
[DIR] barometry/              2022-03-22 22:00    -
[DIR] bulletins/              2022-07-13 13:19    -
[DIR] citypage_weather/       2022-07-13 13:21    -
[DIR] climate/                2020-09-03 16:30    -
[DIR] doc/                    2022-09-28 20:00    -
[DIR] ensemble/               2022-07-13 13:34    -
[DIR] hydrometric/            2021-01-14 14:12    -
[DIR] marine_weather/         2020-12-15 14:51    -
[DIR] meteocode/              2022-07-13 14:01    -
[DIR] model_gdsps/            2021-12-01 21:41    -
[DIR] model_gdwps/            2021-12-01 16:50    -


```

--------------------------------

### RabbitMQ Cluster Setup

Source: https://metpx.github.io/sarracenia/_sources/How2Guides/Admin_Rabbit_Addendum.rst.txt

Configures a RabbitMQ cluster by ensuring the Erlang cookie is consistent across nodes, stopping and starting the application, joining nodes to the cluster, and verifying the cluster status. Requires RabbitMQ server to be stopped on all nodes initially.

```bash
Stop rabbitmq-server on all nodes....

/var/lib/rabbitmq/.erlang.cookie  same on all nodes

on each node restart  /etc/init.d/rabbitmq-server stop/start

on one of the node

rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@"other node"
rabbitmqctl start_app
rabbitmqctl cluster_status
```

--------------------------------

### List Available Configurations

Source: https://metpx.github.io/sarracenia/Explanation/CommandLineGuide.html

Displays all available configuration files, categorized by their location (examples, user plugins, general, user configurations).

```bash
$ subscribe list
```

--------------------------------

### Upload Distribution Package with Twine

Source: https://metpx.github.io/sarracenia/Contribution/Release.html

Use this command to upload your project's distribution package to PyPI. Ensure you have twine installed. This example shows a common error when uploading from Ubuntu 18.04, suggesting a potential version mismatch with system-provided Python packages.

```bash
buntu@canny-tick:~/sr3$ twine upload dist/metpx_sr3-3.0.53rc2-py3-none-any.whl
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.26.18) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Uploading distributions to https://upload.pypi.org/legacy/
Uploading metpx_sr3-3.0.53rc2-py3-none-any.whl
100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 408k/408k [00:00<00:00, 120kB/s]
HTTPError: 400 Client Error: '2.0' is not a valid metadata version. See https://packaging.python.org/specifications/core-metadata for more information. for url: https://upload.pypi.org/legacy/
ubuntu@canny-tick:~/sr3$ 
```

--------------------------------

### Displaying Help Information

Source: https://metpx.github.io/sarracenia/Reference/sr3_post.1.html

Use the `-h` or `--help` option to display the available program options and their descriptions.

```bash
sr3_post -h
```

--------------------------------

### Start a Specific SR3 Configuration

Source: https://metpx.github.io/sarracenia/Tutorials/1_CLI_introduction.html

Use the `!sr3 start <component>/<config>` command to start a specific SR3 configuration. This allows for targeted starting of services, such as a subscribe configuration.

```bash
!sr3 start subscribe/ <config>

```

--------------------------------

### List Sarracenia Examples

Source: https://metpx.github.io/sarracenia/_sources/How2Guides/Email_Ingesting_With_Sarracenia.rst.txt

Lists available Sarracenia examples, useful for finding relevant configuration files like mail.conf.

```bash
fractal% sr3 list examples | grep poll
```

--------------------------------

### Install RabbitMQ Server

Source: https://metpx.github.io/sarracenia/How2Guides/Admin.html

Installs Erlang and the RabbitMQ server on Debian-based systems. Ensure your system is up-to-date before installation.

```bash
apt-get update
apt-get install erlang-nox
apt-get install rabbitmq-server

```

--------------------------------

### Sarracenia Configuration Example

Source: https://metpx.github.io/sarracenia/_modules/sarracenia/config.html

Demonstrates how to configure a sarracenia subscriber instance with broker credentials, topic prefix, component type, and subscription details. Includes settings for download, batch processing, and message limits.

```python
cfg.broker = sarracenia.config.credentials.Credential('amqps://anonymous:anonymous@hpfx.collab.science.gc.ca')
         cfg.topicPrefix = [ 'v02', 'post']
         cfg.component = 'subscribe'
         cfg.config = 'flow_demo'
         cfg.action = 'start'
         cfg.subscriptions= [ { 
             'broker': cfg.broker,
             'bindings': { 'exchange': 'xpublic', 'prefix': ['v02', 'post'], 'sub': [ '*.WXO-DD.observations.swob-ml.#' ] },
             'queue': { 
                 'name' : 'q_anonymous.subscriber_test2', 
                 'template':'q_${BROKER_USER}_${COMPONENT}_${CONFIG}_${HOSTNAME}_Demo123' ,
                 'expire': 600,
                 }
             } ]
         cfg.download=True
         cfg.batch=1
         cfg.messageCountMax=5
       
         # set the instance number for the flow class.
         cfg.no=0
       
       # and at the end call finalize

       cfg.finalize()
```

--------------------------------

### Install AMQP for Debian Package

Source: https://metpx.github.io/sarracenia/Contribution/Development.html

Installs the AMQP library required for Sarracenia when using Debian package installations.

```bash
apt install python3-amqp
```

--------------------------------

### Start Services and Monitor Logs

Source: https://metpx.github.io/sarracenia/Tutorials/Setup_a_local_subscriber.html

Initializes the necessary directories, starts the 'sr3' service, and begins logging for the watcher. This step initiates the file watching and reporting process.

```bash
$ mkdir -p /tmp/sarra/{in,out}put
$ sr3 start
$ sr3 log watch/test-watch

```

--------------------------------

### Example sr_report Configuration and Output

Source: https://metpx.github.io/sarracenia/_sources/How2Guides/Admin.rst.txt

This example shows a configuration file for sr_report and its subsequent output when run in the foreground. It demonstrates active downloading from a pump and message reception.

```bash
blacklab% more boulelog.conf

broker amqps://feeder@boule.example.com/
exchange xreport
accept .*

blacklab%

blacklab% sr_report foreground boulelog.conf 
2016-03-28 16:29:53,721 [INFO] sr_report start
2016-03-28 16:29:53,721 [INFO] sr_report run
2016-03-28 16:29:53,722 [INFO] AMQP  broker(boule.example.com) user(feeder) vhost(/)
2016-03-28 16:29:54,484 [INFO] Binding queue q_feeder.sr_report.boulelog.06413933.71328785 with key v02.report.# from exchange xreport on broker amqps://feeder@boule.example.com/
2016-03-28 16:29:55,732 [INFO] Received notice  20160328202955.139 http://boule.example.com/ radar/CAPPI/GIF/XLA/201603282030_XLA_CAPPI_1.5_RAIN.gif 201 blacklab anonymous -0.040751
2016-03-28 16:29:56,393 [INFO] Received notice  20160328202956.212 http://boule.example.com/ radar/CAPPI/GIF/XMB/201603282030_XMB_CAPPI_1.5_RAIN.gif 201 blacklab anonymous -0.159043
2016-03-28 16:29:56,479 [INFO] Received notice  20160328202956.179 http://boule.example.com/ radar/CAPPI/GIF/XLA/201603282030_XLA_CAPPI_1.0_SNOW.gif 201 blacklab anonymous 0.143819
2016-03-28 16:29:56,561 [INFO] Received notice  20160328202956.528 http://boule.example.com/ radar/CAPPI/GIF/XMB/201603282030_XMB_CAPPI_1.0_SNOW.gif 201 blacklab anonymous -0.119164
2016-03-28 16:29:57,557 [INFO] Received notice  20160328202957.405 http://boule.example.com/ bulletins/alphanumeric/20160328/SN/CWVR/20/SNVD17_CWVR_282000___01910 201 blacklab anonymous -0.161522
2016-03-28 16:29:57,642 [INFO] Received notice  20160328202957.406 http://boule.example.com/ bulletins/alphanumeric/20160328/SN/CWVR/20/SNVD17_CWVR_282000___01911 201 blacklab anonymous -0.089808
2016-03-28 16:29:57,729 [INFO] Received notice  20160328202957.408 http://boule.example.com/ bulletins/alphanumeric/20160328/SN/CWVR/20/SNVD17_CWVR_282000___01912 201 blacklab anonymous -0.043441
2016-03-28 16:29:58,723 [INFO] Received notice  20160328202958.471 http://boule.example.com/ radar/CAPPI/GIF/WKR/201603282030_WKR_CAPPI_1.5_RAIN.gif 201 blacklab anonymous -0.131236
2016-03-28 16:29:59,400 [INFO] signal stop
2016-03-28 16:29:59,400 [INFO] sr_report stop
blacklab%
```

--------------------------------

### Start Subscriber with sr3

Source: https://metpx.github.io/sarracenia/Tutorials/Setup_a_remote_subscriber.html

Use this command to start a subscriber service named 'swob'. Monitor the output for confirmation that the subscription has started.

```bash
$ sr3 start swob
  2015-12-03 06:53:35,268 [INFO] user_config = 0 ../swob.conf
  2015-12-03 06:53:35,269 [INFO] instances 1
  2015-12-03 06:53:35,270 [INFO] sr3 subscribe swob 0001 started

```

--------------------------------

### Log Start Event

Source: https://metpx.github.io/sarracenia/_modules/sarracenia/flowcb/log.html

Logs 'starting' and statistics if 'on_start' is enabled in logEvents. Calls the stats method to report current metrics upon starting.

```python
def on_start(self):
        if set(['on_start']) & self.o.logEvents:
            self.stats()
            logger.info("starting")
```

--------------------------------

### on_start Callback

Source: https://metpx.github.io/sarracenia/Reference/flowcb.html

This callback is executed after a connection is established with the broker and before any message transfer occurs. It is used for initial setup.

```python
def on_start(self) -> None:
```
After the connection is established with the broker and things are instantiated, but
before any message transfer occurs.

```
```

--------------------------------

### Generating Windows Installer

Source: https://metpx.github.io/sarracenia/_sources/Contribution/Release.rst.txt

Generates a Windows installer using pynsist. This script requires Python 3.10 or higher and NSIS to be installed. The output is logged to 'log.txt'.

```bash
# if not already installed:
#   sudo apt install nsis
#   pip3 install pynsist wheel

git checkout pre-release
./generate-win-installer.sh 3.10.11 2>&1 > log.txt
```

--------------------------------

### Example AMQP/AMQPS Credentials

Source: https://metpx.github.io/sarracenia/Reference/sr3_credentials.7.html

Examples of how to format credentials for AMQP and AMQPS connections, including user, password, host, and optional vhost.

```plaintext
amqp://user1:password1@host/
```

```plaintext
amqps://user2:password2@host:5671/dev
```

```plaintext
amqps://usern:passwd@host/ login_method=PLAIN
```

--------------------------------

### Build and Install Debian Package Locally

Source: https://metpx.github.io/sarracenia/Contribution/Development.html

Builds a Debian package for local installation and testing. This command needs to be re-run after each code change and requires python3-amqp to be installed.

```bash
dpkg -i
```

--------------------------------

### Option Order and Scope Example 1

Source: https://metpx.github.io/sarracenia/_sources/Reference/sr3_options.7.rst.txt

Demonstrates how 'mirror', 'directory', and 'accept' options affect file processing based on their order. In this sequence, 'mirror' is off.

```plaintext
mirror off
directory /data/just_flat_files_here_please
accept .*flatones.*
```

--------------------------------

### Enable and Start User Systemd Service

Source: https://metpx.github.io/sarracenia/How2Guides/Admin.html

Commands to enable and start the Metpx user service for systemd-based systems. Ensure the user service is enabled to start on every login.

```bash
systemctl --user enable metpx-sr3
systemctl --user start metpx-sr3
```

--------------------------------

### Example MQTT Subscription Topic Construction

Source: https://metpx.github.io/sarracenia/_sources/Reference/sr3_options.7.rst.txt

Demonstrates how sarracenia constructs MQTT subscription topics using queueName, exchange, topicPrefix, and subtopic. This example shows the default behavior.

```text
broker mqtt://user@broker
queueName q_user_Doreen
subtopic #

resulting subscription topic:   $share/q_user_Doreen/xpublic/v03/#
```
Tokens:
Raw