### List Example Configurations with sr3

Source: https://metpx.github.io/sarracenia/Explanation/CommandLineGuide.html

Use 'sr3 list examples' to view included configuration templates that can be used as starting points for new configurations.

```bash
$ sr3 list examples
  Sample Configurations: (from: /home/peter/Sarracenia/development/sarra/examples )
  cpump/cno_trouble_f00.inc        poll/aws-nexrad.conf             poll/pollingest.conf
  poll/pollnoaa.conf               poll/pollsoapshc.conf            poll/pollusgs.conf
  poll/pulse.conf                  post/WMO_mesh_post.conf          sarra/wmo_mesh.conf
  sender/ec2collab.conf            sender/pitcher_push.conf         shovel/no_trouble_f00.inc
  subscribe/WMO_Sketch_2mqtt.conf  subscribe/WMO_Sketch_2v3.conf    subscribe/WMO_mesh_CMC.conf
  subscribe/WMO_mesh_Peer.conf     subscribe/aws-nexrad.conf        subscribe/dd_2mqtt.conf
  subscribe/dd_all.conf            subscribe/dd_amis.conf           subscribe/dd_aqhi.conf
  subscribe/dd_cacn_bulletins.conf subscribe/dd_citypage.conf       subscribe/dd_cmml.conf
  subscribe/dd_gdps.conf           subscribe/dd_ping.conf           subscribe/dd_radar.conf
  subscribe/dd_rdps.conf           subscribe/dd_swob.conf           subscribe/ddc_cap-xml.conf
  subscribe/ddc_normal.conf        subscribe/downloademail.conf     subscribe/ec_ninjo-a.conf
  subscribe/hpfx_amis.conf         subscribe/local_sub.conf         subscribe/pitcher_pull.conf
  subscribe/sci2ec.conf            subscribe/subnoaa.conf           subscribe/subsoapshc.conf
  subscribe/subusgs.conf           watch/master.conf                watch/pitcher_client.conf
  watch/pitcher_server.conf        watch/sci2ec.conf


```

--------------------------------

### Install and Configure RabbitMQ

Source: https://metpx.github.io/sarracenia/Tutorials/Setup_a_local_subscriber.html

Installs RabbitMQ server, lists users, adds a new user 'bob' with a password, and lists virtual hosts. This is the initial setup for the local broker.

```bash
$ sudo apt install rabbitmq-server
$ sudo rabbitmqctl list_users
  Listing users ...
  user    tags
  guest   [administrator]

$ sudo rabbitmqctl add_user 'bob'
  Adding user "bob" ...
  Password: robert

$ sudo rabbitmqctl list_vhosts
  Listing vhosts ...
  name
  /

```

--------------------------------

### Start Message Publishers

Source: https://metpx.github.io/sarracenia/_modules/sarracenia/flowcb/post/message.html

Initiates the setup process for all message publishers. This method should be called when the service starts to ensure publishers are ready to send messages.

```python
    def on_start(self):
        for p in self.posters:
            if hasattr(p,'putSetup'):
                p.putSetup()

        logger.debug('starting')
```

--------------------------------

### Example Subscriber Configuration for Troubleshooting

Source: https://metpx.github.io/sarracenia/_sources/How2Guides/subscriber.rst.txt

This example demonstrates how to create a sample subscriber configuration to identify slow downloads by examining lag in the download log. It's a starting point for diagnosing performance issues.

```shell
fractal% sr3 list ie

 Sample Configurations: (from: /home/peter/Sarracenia/sr3/sarracenia/examples )
 cpump/cno_trouble_f00.inc        flow/amserver.conf               flow/poll.inc                    flow/post.inc                    flow/report.inc                  flow/sarra.inc                   
 flow/sender.inc                  flow/shovel.inc                  flow/subscribe.inc               flow/watch.inc                   flow/winnow.inc                  poll/airnow.conf                 
 poll/aws-nexrad.conf             poll/mail.conf                   poll/nasa-mls-nrt.conf           poll/noaa.conf                   poll/soapshc.conf                poll/usgs.conf                   
 post/WMO_mesh_post.conf          sarra/wmo_mesh.conf              sender/am_send.conf              sender/ec2collab.conf            sender/pitcher_push.conf         shovel/no_trouble_f00.inc        
 subscribe/aws-nexrad.conf        subscribe/dd_2mqtt.conf          subscribe/dd_all.conf            subscribe/dd_amis.conf           subscribe/dd_aqhi.conf           subscribe/dd_cacn_bulletins.conf 
 subscribe/dd_citypage.conf       subscribe/dd_cmml.conf           subscribe/dd_gdps.conf           subscribe/dd_radar.conf          subscribe/dd_rdps.conf           subscribe/dd_swob.conf           
 subscribe/ddc_cap-xml.conf       subscribe/ddc_normal.conf        subscribe/downloademail.conf     subscribe/ec_ninjo-a.conf        subscribe/hpfxWIS2DownloadAll.conf subscribe/hpfx_amis.conf         
 subscribe/hpfx_citypage.conf     subscribe/local_sub.conf         subscribe/ping.conf              subscribe/pitcher_pull.conf      subscribe/sci2ec.conf            subscribe/subnoaa.conf           
 subscribe/subsoapshc.conf        subscribe/subusgs.conf           sender/am_send.conf              sender/ec2collab.conf            sender/pitcher_push.conf         watch/master.conf                
 watch/pitcher_client.conf        watch/pitcher_server.conf        watch/sci2ec.conf                
fractal% 

pick one ane add it local configuration::
```

--------------------------------

### List Available Sarracenia Configuration Examples

Source: https://metpx.github.io/sarracenia/_sources/Tutorials/1_CLI_introduction.ipynb.txt

Lists all available sample configuration files for Sarracenia, organized by type (e.g., flow, poll, subscribe). These examples can be used as a starting point for defining custom data flows.

```bash
sr3
```

--------------------------------

### Command Line Execution Style Guide

Source: https://metpx.github.io/sarracenia/_sources/Contribution/Documentation.rst.txt

Demonstrates the standard format for writing command-line execution examples in documentation. Includes an initial comment, commands prefixed with '$', and indented relevant output.

```bash
An initial comment describing the following steps or processes::

    $ command 1
      relevant output
    $command 2
      .
      .
      relevant output
      newline relevant output
```

--------------------------------

### Start All Sarracenia Components

Source: https://metpx.github.io/sarracenia/_sources/Explanation/CommandLineGuide.rst.txt

Launches all configured components by executing the 'start' command. This includes gathering global state before starting.

```bash
$ sr3 start
    gathering global state: procs, configs, state files, logs, analysis - Done. 
    starting...Done
```

--------------------------------

### Python Logging Setup

Source: https://metpx.github.io/sarracenia/Tutorials/2_CLI_with_flowcb_demo.html

Demonstrates the standard Pythonic way to set up and use the built-in logging module. Import the module and get a logger instance for the current file.

```python
import logging

logger = logging.getLogger(__name__)

logger.debug('got here')
```

--------------------------------

### Install All Extras via Pip

Source: https://metpx.github.io/sarracenia/Tutorials/Install.html

Install metpx-sr3 with all available extras using the 'all' shortcut with pip.

```bash
$ pip install metpx-sr3[all]
```

--------------------------------

### Sr3 CLI: Start Subscribe Configuration

Source: https://metpx.github.io/sarracenia/How2Guides/UPGRADING.html

In Sr3, the command-line interface uses a single entry point 'sr3'. This example shows the pattern for starting a subscription configuration.

```bash
sr3 start subscribe/config
```

--------------------------------

### Start All Active SR3 Configurations

Source: https://metpx.github.io/sarracenia/Tutorials/1_CLI_introduction.html

Use the `!sr3 start` command to start all active configurations across all SR3 components. This is a convenient way to bring up the entire SR3 system.

```bash
!sr3 start

```

--------------------------------

### Sarracenia Flow Configuration Example

Source: https://metpx.github.io/sarracenia/_sources/Tutorials/3_api_flow_demo.ipynb.txt

Illustrates the initialization of Sarracenia's configuration with various options related to message processing, retry mechanisms, and logging. This setup is typically part of launching a Sarracenia flow.

```python
cfg = sarracenia.config.Config()
cfg.add_option('messageCountMax', 1000000, 'max messages to process')
cfg.add_option('retry_queues', ['work_retry_00'], 'retry queues')
cfg.add_option('retry_driver', 'disk', 'retry driver type')
cfg.add_option('retry_refilter', False, 'refilter retried messages')
cfg.add_option('MemoryMax', 0, 'max memory to use')
cfg.add_option('MemoryBaseLineFile', 100, 'baseline memory file size')
cfg.add_option('MemoryMultiplier', 3, 'memory multiplier')
cfg.add_option('logEvents', {'after_post', 'after_work', 'on_housekeeping', 'after_accept'}, 'events to log')
cfg.add_option('logMessageDump', False, 'log message dumps')

# masks are a compiled form of accept/reject directives.
# a relPath is compared to the regex in the mask.
# If the regex matches, and accept is true, then the message is accepted for further processing.
# If the regex matches, but accept is False, then processing of the message is stopped (the message is rejected.)
# masks are a tuple. the meaning can be looked up in the sr3(1) man page.
#
# *  pattern_string,      the input regular expression string, to be compiled by re routines.
# *  directory,           where to put the files downloaded (root of the tree, when mirroring)
# *  fn,                  transformation of filename to do. None is the 99% use case.
# *  regex,               compiled regex version of the pattern_string
# *  accept(True/False),  if pattern matches then accept message for further processing.
# *  mirror(True/False),  when downloading build a complete tree to mirror the source, or just dump in directory
# *  strip(True/False),   modify the relpath by stripping entries from the left.
# *  pstrip(True/False),  strip entries based on patterm
# *  flatten(char ... '/' means do not flatten.) )

# These settings are needed because they would ordinarily be set by the sarracenia.instance class which is
# normally used to launch flows. They allow setting up of run-time paths for retry_queues, and statefiles,
# to remember settings if need be between runs.
cfg.add_option('no', '/tmp/sarracenia_demo/no', 'path for no messages')
cfg.add_option('pid_filename', '/tmp/sarracenia_demo/pid', 'pid filename')

# Example of a mask
# cfg.masks = [
#     ('^.*$', '/', None, None, True, False, False, False, '/'),
# ]

```

--------------------------------

### Start All Sarracenia Components

Source: https://metpx.github.io/sarracenia/How2Guides/Admin.html

Use this command to start all components with named configurations. Ensure configurations for sarra, subscribe, winnow, and log are set up.

```bash
sr3 start

```

--------------------------------

### Minimal Sarracenia Configuration Options

Source: https://metpx.github.io/sarracenia/_modules/sarracenia.html

Example of a minimal options dictionary for Sarracenia consumer setup. It includes broker credentials and topic prefixes.

```python
options = sarracenia.config.no_file_config()
options['broker'] = sarracenia.config.credentials.Credential(
          'amqps://anonymous:anonymous@hpfx.collab.science.gc.ca')
options['topicPrefix'] = [ 'v02', 'post' ]
options['subscriptions'] = sarracenia.config.subscription.Subscriptions( [ 
               sarracenia.config.subscription.Subscriptions( 
                  options, 
                  options['queueName'],
                  'q_anonymous_' + socket.getfqdn() + '_SomethingHelpfulToYou',
                  '#' 
               ) ] )
```

--------------------------------

### Generate Windows Installer on Ubuntu

Source: https://metpx.github.io/sarracenia/Contribution/Release.html

Commands to set up the environment and generate a Windows installer on an Ubuntu system. Requires NSIS, pynsist, and wheel to be installed. Uses a specific Python version and redirects output to a log file.

```bash
# if not already installed:
sudo apt install nsis
pip3 install pynsist wheel

git checkout pre-release
./generate-win-installer.sh 3.10.11 2>&1 > log.txt
```

--------------------------------

### Start a Specific SR3 Configuration

Source: https://metpx.github.io/sarracenia/Tutorials/1_CLI_introduction.html

Use the `!sr3 start <component>/<config>` command to start a specific SR3 configuration. This allows for targeted starting of services, such as a subscribe configuration.

```bash
!sr3 start subscribe/ <config>

```

--------------------------------

### List Sarracenia Examples

Source: https://metpx.github.io/sarracenia/_sources/How2Guides/Email_Ingesting_With_Sarracenia.rst.txt

Lists available Sarracenia examples, useful for finding relevant configuration files like mail.conf.

```bash
fractal% sr3 list examples | grep poll
```

--------------------------------

### Example AMQP/AMQPS Credentials

Source: https://metpx.github.io/sarracenia/Reference/sr3_credentials.7.html

Examples of how to format credentials for AMQP and AMQPS connections, including user, password, host, and optional vhost.

```plaintext
amqp://user1:password1@host/
```

```plaintext
amqps://user2:password2@host:5671/dev
```

```plaintext
amqps://usern:passwd@host/ login_method=PLAIN
```

--------------------------------

### SFTP Credentials Examples

Source: https://metpx.github.io/sarracenia/Explanation/CommandLineGuide.html

Examples of how to format SFTP credentials, including username, password, host, and optional SSH key file.

```text
sftp://user5:password5@host
```

```text
sftp://user6:password6@host:22 ssh_keyfile=/users/local/.ssh/id_dsa
```

--------------------------------

### Start RabbitMQ Server (Ubuntu)

Source: https://metpx.github.io/sarracenia/How2Guides/Admin.html

Use this command to start the RabbitMQ server on older Ubuntu systems.

```bash
service rabbitmq-server start

```

--------------------------------

### Example Download URL

Source: https://metpx.github.io/sarracenia/Reference/sr_post.7.html

Provides an example of a complete URL that consumers can use to download data.

```text
sftp://afsiext@cmcdataserver/data/NRPDS/outputs/NRPDS_HiRes_000.gif
```

--------------------------------

### sr_watch Configuration Example

Source: https://metpx.github.io/sarracenia/_sources/Contribution/on_part_assembly.rst.txt

Example configuration for sr_watch to monitor a directory for incoming partition files.

```ini
- path <path of directory to watch>
- on_part /usr/lib/python3/dist-packages/sarra/plugins/part_file_assemble.py
- accept *.Part
- accept_unmatch False # Makes it only acccept the pattern above
```

--------------------------------

### Example of flow_setup.sh execution

Source: https://metpx.github.io/sarracenia/Contribution/Development.html

Demonstrates the output of the flow_setup.sh script, including log cleaning messages.

```bash
blacklab% ./flow_setup.sh
cleaning logs, just in case
rm: cannot remove '/home/peter/.cache/sarra/log/*': No such file or directory
```

--------------------------------

### FTP Credentials Examples

Source: https://metpx.github.io/sarracenia/Explanation/CommandLineGuide.html

Examples of how to format FTP credentials, including username, password, host, and connection mode (passive/active, binary/ascii).

```text
ftp://user7:password7@host passive,binary
```

```text
ftp://user8:password8@host:2121 active,ascii
```

--------------------------------

### Instance Startup and Configuration Loading

Source: https://metpx.github.io/sarracenia/_modules/sarracenia/instance.html

Handles instance startup, including setting up logging, permissions, and loading configuration. It prepares the environment by setting environment variables declared in the configuration and then initializes and runs the core instance.

```python
                if hasattr(cfg_preparse, 'permLog'):
                    os.chmod(logfilename, cfg_preparse.permLog)

                # FIXME: https://docs.python.org/3/library/contextlib.html portable redirection...
                if sys.platform != 'win32':
                    os.dup2(handler.stream.fileno(), 1)
                    os.dup2(handler.stream.fileno(), 2)

        else:
            logger.setLevel(logLevel)

        signal.signal(signal.SIGTERM, self.stop_signal)
        signal.signal(signal.SIGINT, self.stop_signal)

        if cfg_preparse.statehost:
            hostdir = cfg_preparse.hostdir
        else:
            hostdir = None

        if not os.path.isdir(os.path.dirname(pidfilename)):
            pathlib.Path(os.path.dirname(pidfilename)).mkdir(parents=True, exist_ok=True)

        with open(pidfilename, 'w') as pfn:
            pfn.write('%d' % os.getpid())

        cfg = sarracenia.config.one_config(component, config, cfg_preparse.action, isPost=False, hostDir=hostdir)

        cfg.novipFilename = pidfilename.replace(".pid", ".noVip")

        if not hasattr(cfg, 'env_declared'):
            sys.exit(0)

        for n in cfg.env_declared:
            os.environ[n] = cfg.env[n]
            os.putenv(n, cfg.env[n])

        self.o = cfg
        self.running_instance = Flow.factory(cfg)

        self.running_instance.run()

        if os.path.isfile(pidfilename):
            os.unlink(pidfilename)
        sys.exit(0)
```

--------------------------------

### Install Sarrac C Client using PPA

Source: https://metpx.github.io/sarracenia/_sources/Tutorials/Install.rst.txt

Install the Sarrac C client on Ubuntu using a PPA. This method ensures you get prebuilt binaries and necessary dependencies.

```bash
$ sudo add-apt-repository ppa:ssc-hpc-chp-spc/metpx
$ sudo apt-get update
$ sudo apt-get install metpx-sr3c
```

--------------------------------

### List Available SR3 Examples

Source: https://metpx.github.io/sarracenia/Tutorials/1_CLI_introduction.html

Lists all available example configurations for Sarracenia flows. This helps in identifying suitable configurations for various tasks.

```bash
!sr3 list examples
```

--------------------------------

### Command Line Syntax Comparison v2 vs v3

Source: https://metpx.github.io/sarracenia/How2Guides/Plugins_v2ToSr3.html

Illustrates the difference in command-line entry points for starting components between Sarracenia v2 and v3.

```bash
v2:  sr_*component* start config

v3:  sr3 start *component*/config

```

--------------------------------

### Upload to PyPI using setup.py (Deprecated)

Source: https://metpx.github.io/sarracenia/_sources/Contribution/Release.rst.txt

This command was used on older systems to upload a new release to PyPI. It is now deprecated.

```bash
python3 setup.py bdist_wheel upload
```

--------------------------------

### Regex Group Compatibility: Sundew Groups Start from 0 (True)

Source: https://metpx.github.io/sarracenia/_sources/Reference/sr3_options.7.rst.txt

When 'sundew_compat_regex_first_match_is_zero' is True, Sundew group numbering (starting from 0) is used, differing from Python's convention where group 0 is the entire string and subsequent groups start from 1. This example shows the resulting directory structure.

```text
* sundew_compat_regex_first_match_is_zero: True
* input url:  https://hpfx.collab.science.gc.ca/20231127/WXO-DD/meteocode/que/cmml/TRANSMIT.FPCN71.11.27.1000Z.xml
* accept pattern: .*/WXO-DD/meteocode/(atl|ont|pnr|pyr|que)/.*/TRANSMIT\.FP([A-Z][A-Z]).*([0-2][0-9][0-6][0-9]Z).*
* directory setting: /tmp/meteocode/${2}/${0}/${1}
* resulting directory:  /tmp/meteocode/1000Z/que/CN
```

--------------------------------

### AMQP Topic Structure Example

Source: https://metpx.github.io/sarracenia/Reference/sr_post.7.html

Illustrates the complete topic structure for a notification message, starting with the protocol version and file path.

```text
v03.a.b.c.d
```

--------------------------------

### Start All Components

Source: https://metpx.github.io/sarracenia/Explanation/CommandLineGuide.html

The 'start' command launches all configured components of the sarracenia project. It first gathers global state including processes, configurations, state files, logs, and analysis before initiating the startup sequence.

```bash
$ sr3 start
  gathering global state: procs, configs, state files, logs, analysis - Done.
  starting...Done

```

--------------------------------

### Sr3 CLI: Multiple Configurations

Source: https://metpx.github.io/sarracenia/How2Guides/UPGRADING.html

Sr3 allows specifying multiple configurations in a single command. This example demonstrates starting poll, subscribe, and sender configurations simultaneously.

```bash
sr3 start poll/airnow subscribe/airnow sender/cmqb
```

--------------------------------

### Add a new configuration from an example

Source: https://metpx.github.io/sarracenia/_sources/Explanation/CommandLineGuide.rst.txt

The 'sr3 add' command copies a configuration template from the examples directory to the user's configuration directory. The output shows the source and destination paths.

```bash
$ sr3 add dd_all.conf
    add: 2021-01-24 18:04:57,018 [INFO] sarracenia.sr add copying: /usr/lib/python3/dist-packages/sarracenia/examples/subscribe/dd_all.conf to /home/peter/.config/sr3/subscribe/dd_all.conf 
```

--------------------------------

### Regex Group Compatibility: Python Groups Start from 1 (False)

Source: https://metpx.github.io/sarracenia/_sources/Reference/sr3_options.7.rst.txt

When 'sundew_compat_regex_first_match_is_zero' is False, the system conforms to Python's regular expression group numbering, where group 0 is the entire string and subsequent groups start from 1. This example shows the directory setting required to achieve the same result as the previous snippet.

```text
* sundew_compat_regex_first_match_is_zero: False
* directory setting: /tmp/meteocode/${3}/${1}/${2}
* to get the same result.
```

--------------------------------

### Output of Flow Setup Script

Source: https://metpx.github.io/sarracenia/_sources/Contribution/Development.rst.txt

This is the typical output observed when running the flow_setup.sh script, indicating the cleaning of logs and the copying of various configuration and example files for the flow test environment.

```bash
blacklab% ./flow_setup.sh
cleaning logs, just in case
rm: cannot remove '/home/peter/.cache/sarra/log/*': No such file or directory
Adding flow test configurations...
2018-02-10 14:22:58,944 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpump/cno_trouble_f00.inc to /home/peter/.config/sarra/cpump/cno_trouble_f00.inc.
2018-02-10 09:22:59,204 [INFO] copying /home/peter/src/sarracenia/sarra/examples/shovel/no_trouble_f00.inc to /home/peter/.config/sarra/shovel/no_trouble_f00.inc
2018-02-10 14:22:59,206 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpost/veille_f34.conf to /home/peter/.config/sarra/cpost/veille_f34.conf.
2018-02-10 14:22:59,207 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpump/pelle_dd1_f04.conf to /home/peter/.config/sarra/cpump/pelle_dd1_f04.conf.
2018-02-10 14:22:59,208 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpump/pelle_dd2_f05.conf to /home/peter/.config/sarra/cpump/pelle_dd2_f05.conf.
2018-02-10 14:22:59,208 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpump/xvan_f14.conf to /home/peter/.config/sarra/cpump/xvan_f14.conf.
2018-02-10 14:22:59,209 [INFO] copying /usr/lib/python3/dist-packages/sarra/examples/cpump/xvan_f15.conf to /home/peter/.config/sarra/cpump/xvan_f15.conf.
2018-02-10 09:22:59,483 [INFO] copying /home/peter/src/sarracenia/sarra/examples/poll/f62.conf to /home/peter/.config/sarra/poll/f62.conf
2018-02-10 09:22:59,756 [INFO] copying /home/peter/src/sarracenia/sarra/examples/post/shim_f63.conf to /home/peter/.config/sarra/post/shim_f63.conf
2018-02-10 09:23:00,030 [INFO] copying /home/peter/src/sarracenia/sarra/examples/post/test2_f61.conf to /home/peter/.config/sarra/post/test2_f1.conf
2018-02-10 09:23:00,299 [INFO] copying /home/peter/src/sarracenia/sarra/examples/report/tsarra_f20.conf to /home/peter/.config/sarra/report/tsarra_f20.conf
2018-02-10 09:23:00,561 [INFO] copying /home/peter/src/sarracenia/sarra/examples/report/twinnow00_f10.conf to /home/peter/.config/sarra/report/twinnow00_f10.conf
```

--------------------------------

### List Available Configurations with sr3

Source: https://metpx.github.io/sarracenia/How2Guides/subscriber.html

Use 'sr3 list' to see all available sample and user configurations. This command helps in discovering pre-defined setups.

```bash
$ sr3 list examples
```

--------------------------------

### On Start Entry Point

Source: https://metpx.github.io/sarracenia/_sources/How2Guides/FlowCallbacks.rst.txt

Called after connection establishment and instantiation, but before any notification message transfer.

```python
def on_start(self):
         After the connection is established with the broker and things are instantiated, but
         before any notification message transfer occurs.
```

--------------------------------

### Get detailed information for a specific configuration

Source: https://metpx.github.io/sarracenia/_sources/Explanation/CommandLineGuide.rst.txt

When 'sr3 status' indicates a configuration is partial, use 'sr3 dump' with 'grep' to retrieve detailed information, including the command line used to start it.

```bash
$ sr3 dump | grep sender/tsource2send_f50
    49308: name:sr3_sender.py cmdline:['/usr/bin/python3', '/usr/lib/python3/dist-packages/sarracenia/instance.py', '--no', '1', 'start', 'sender/tsource2send_f50']
    q_tsource.sr_sender.tsource2send_f50.58710892.12372870: ['sender/tsource2send_f50']
```

--------------------------------

### LongFlow plugin example for self-test

Source: https://metpx.github.io/sarracenia/Reference/flowcb.html

This plugin is for self-test purposes. It creates a header 'toolong' longer than 255 characters, which gets truncated. It helps test truncation handling and associated warning messages.

```python
class sarracenia.flowcb.accept.longflow.LongFlow(_options_)[source]:
    
    Bases: `FlowCB` 

    __init__(_options_)[source]
```

--------------------------------

### Install Dependencies and Build Documentation Locally

Source: https://metpx.github.io/sarracenia/_sources/Contribution/Documentation.rst.txt

Install development requirements and build the Sphinx documentation locally. Navigate to the docs directory and run 'make html' to generate the output.

```bash
pip install -f requirements-dev.txt
cd docs
make html
```

--------------------------------

### Example Configuration for Standard Format Tree

Source: https://metpx.github.io/sarracenia/Reference/sr3_options.7.html

This configuration combines mirror, sourceFromExchange, and directory settings to ensure data arrives in the standard format tree. It's a common setup for structured data ingestion.

```plaintext
*mirror true*
*sourceFromExchange true*
*directory ${PBD}/${YYYYMMDD}/${SOURCE}*
```

--------------------------------

### Install Sarracenia using pip (with Extras)

Source: https://metpx.github.io/sarracenia/_sources/Tutorials/Install.rst.txt

Command to install Sarracenia with additional features like AMQP, MQTT, VIP, FTP poll, and file type support. Use 'all' for all available extras.

```bash
pip install metpx-sr3[amqp,mqtt,vip,ftppoll,filetype]
```

--------------------------------

### Initialize and Configure Sarracenia

Source: https://metpx.github.io/sarracenia/api-documentation.html

Demonstrates how to initialize an empty configuration and manually set various parameters including broker credentials, topic prefixes, component settings, and subscription details. This is useful for setting up a custom Sarracenia configuration programmatically.

```python
cfg = no_file_config()

cfg.broker = sarracenia.config.credentials.Credential('amqps://anonymous:anonymous@hpfx.collab.science.gc.ca')
cfg.topicPrefix = [ 'v02', 'post']
cfg.component = 'subscribe'
cfg.config = 'flow_demo'
cfg.action = 'start'
cfg.subscriptions= [ { 
    'broker': cfg.broker,
    'bindings': { 'exchange': 'xpublic', 'prefix': ['v02', 'post'], 'sub': [ '*.WXO-DD.observations.swob-ml.#' ] },
    'queue': {
        'name' : 'q_anonymous.subscriber_test2',
        'template':'q_${BROKER_USER}_${COMPONENT}_${CONFIG}_${HOSTNAME}_Demo123' ,
        'expire': 600,
        }
    } ]
cfg.download=True
cfg.batch=1
cfg.messageCountMax=5

# set the instance number for the flow class.
cfg.no=0


```

--------------------------------

### Read and Write Data Chunk

Source: https://metpx.github.io/sarracenia/_modules/sarracenia/transfer.html

Resets transfer statistics such as total bytes transferred, start time, and last log time before initiating a read/write operation. This method is a setup for transferring data between source and destination.

```python
    def read_write(self, src, dst, length=0):
        logger.debug("sr_proto read_write")

        # reset speed

        rw_length = 0
        self.tbytes = 0.0
        self.tbegin = nowflt()
        self.lastLog = self.tbegin
```

--------------------------------

### RabbitMQ Cluster Setup

Source: https://metpx.github.io/sarracenia/_sources/How2Guides/Admin_Rabbit_Addendum.rst.txt

Configures a RabbitMQ cluster by ensuring the Erlang cookie is consistent across nodes, stopping and starting the application, joining nodes to the cluster, and verifying the cluster status. Requires RabbitMQ server to be stopped on all nodes initially.

```bash
Stop rabbitmq-server on all nodes....

/var/lib/rabbitmq/.erlang.cookie  same on all nodes

on each node restart  /etc/init.d/rabbitmq-server stop/start

on one of the node

rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@"other node"
rabbitmqctl start_app
rabbitmqctl cluster_status
```

--------------------------------

### Build and Install RPM Package

Source: https://metpx.github.io/sarracenia/Tutorials/Install.html

Clone the Sarracenia repository, navigate into the directory, build an RPM package using setuptools, and then install the generated RPM file.

```bash
$ git clone https://github.com/MetPX/sarracenia
$ cd sarracenia

$ python3 setup.py bdist_rpm
$ sudo rpm -i dist/*.noarch.rpm
```

--------------------------------

### Install and Configure RabbitMQ Server

Source: https://metpx.github.io/sarracenia/_sources/Tutorials/Setup_a_local_subscriber.rst.txt

Installs RabbitMQ server and configures a new user 'bob' with a password. Sets permissions for the user on the default vhost and enables management plugins.

```bash
$ sudo apt install rabbitmq-server
$ sudo rabbitmqctl list_users
    Listing users ...
    user    tags
    guest   [administrator]

$ sudo rabbitmqctl add_user 'bob'
    Adding user "bob" ...
    Password: robert

$ sudo rabbitmqctl list_vhosts
    Listing vhosts ...
    name
    /

$ sudo rabbitmqctl set_permissions -p "/" "bob" ".*" ".*" ".*"
    Settting permissions for user "bob" in vhost "/" ...

$ sudo rabbitmqctl set_user_tags bob management
    Setting tags for user "bob" to [management] ...

$ sudo rabbitmq-plugins enable rabbitmq_management
$ systemctl restart rabbitmq-server
```

--------------------------------

### Upload Distribution Package with Twine

Source: https://metpx.github.io/sarracenia/Contribution/Release.html

Use this command to upload your project's distribution package to PyPI. Ensure you have twine installed. This example shows a common error when uploading from Ubuntu 18.04, suggesting a potential version mismatch with system-provided Python packages.

```bash
buntu@canny-tick:~/sr3$ twine upload dist/metpx_sr3-3.0.53rc2-py3-none-any.whl
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.26.18) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Uploading distributions to https://upload.pypi.org/legacy/
Uploading metpx_sr3-3.0.53rc2-py3-none-any.whl
100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 408k/408k [00:00<00:00, 120kB/s]
HTTPError: 400 Client Error: '2.0' is not a valid metadata version. See https://packaging.python.org/specifications/core-metadata for more information. for url: https://upload.pypi.org/legacy/
ubuntu@canny-tick:~/sr3$ 
```

--------------------------------

### Example Data Products from a Source

Source: https://metpx.github.io/sarracenia/How2Guides/subscriber.html

This snippet illustrates the organization of data products under a specific source directory on a Sarracenia data pump.

```html
Icon  Name                    Last modified      Size  Description
[TXT] about_dd_apropos.txt    2021-05-17 13:23   1.0K
[DIR] air_quality/            2020-12-10 14:47    -
[DIR] alerts/                 2022-07-13 12:00    -
[DIR] analysis/               2022-07-13 13:17    -
[DIR] barometry/              2022-03-22 22:00    -
[DIR] bulletins/              2022-07-13 13:19    -
[DIR] citypage_weather/       2022-07-13 13:21    -
[DIR] climate/                2020-09-03 16:30    -
[DIR] doc/                    2022-09-28 20:00    -
[DIR] ensemble/               2022-07-13 13:34    -
[DIR] hydrometric/            2021-01-14 14:12    -
[DIR] marine_weather/         2020-12-15 14:51    -
[DIR] meteocode/              2022-07-13 14:01    -
[DIR] model_gdsps/            2021-12-01 21:41    -
[DIR] model_gdwps/            2021-12-01 16:50    -


```

--------------------------------

### Install RabbitMQ Server

Source: https://metpx.github.io/sarracenia/How2Guides/Admin.html

Installs Erlang and the RabbitMQ server on Debian-based systems. Ensure your system is up-to-date before installation.

```bash
apt-get update
apt-get install erlang-nox
apt-get install rabbitmq-server

```

--------------------------------

### List Available Configurations

Source: https://metpx.github.io/sarracenia/Explanation/CommandLineGuide.html

Displays all available configuration files, categorized by their location (examples, user plugins, general, user configurations).

```bash
$ subscribe list
```

--------------------------------

### Install AMQP for Debian Package

Source: https://metpx.github.io/sarracenia/Contribution/Development.html

Installs the AMQP library required for Sarracenia when using Debian package installations.

```bash
apt install python3-amqp
```

--------------------------------

### Displaying Help Information

Source: https://metpx.github.io/sarracenia/Reference/sr3_post.1.html

Use the `-h` or `--help` option to display the available program options and their descriptions.

```bash
sr3_post -h
```

--------------------------------

### Start Subscriber with sr3

Source: https://metpx.github.io/sarracenia/Tutorials/Setup_a_remote_subscriber.html

Use this command to start a subscriber service named 'swob'. Monitor the output for confirmation that the subscription has started.

```bash
$ sr3 start swob
  2015-12-03 06:53:35,268 [INFO] user_config = 0 ../swob.conf
  2015-12-03 06:53:35,269 [INFO] instances 1
  2015-12-03 06:53:35,270 [INFO] sr3 subscribe swob 0001 started

```

--------------------------------

### Sarracenia Configuration Example

Source: https://metpx.github.io/sarracenia/_modules/sarracenia/config.html

Demonstrates how to configure a sarracenia subscriber instance with broker credentials, topic prefix, component type, and subscription details. Includes settings for download, batch processing, and message limits.

```python
cfg.broker = sarracenia.config.credentials.Credential('amqps://anonymous:anonymous@hpfx.collab.science.gc.ca')
         cfg.topicPrefix = [ 'v02', 'post']
         cfg.component = 'subscribe'
         cfg.config = 'flow_demo'
         cfg.action = 'start'
         cfg.subscriptions= [ { 
             'broker': cfg.broker,
             'bindings': { 'exchange': 'xpublic', 'prefix': ['v02', 'post'], 'sub': [ '*.WXO-DD.observations.swob-ml.#' ] },
             'queue': { 
                 'name' : 'q_anonymous.subscriber_test2', 
                 'template':'q_${BROKER_USER}_${COMPONENT}_${CONFIG}_${HOSTNAME}_Demo123' ,
                 'expire': 600,
                 }
             } ]
         cfg.download=True
         cfg.batch=1
         cfg.messageCountMax=5
       
         # set the instance number for the flow class.
         cfg.no=0
       
       # and at the end call finalize

       cfg.finalize()
```

--------------------------------

### Log Start Event

Source: https://metpx.github.io/sarracenia/_modules/sarracenia/flowcb/log.html

Logs 'starting' and statistics if 'on_start' is enabled in logEvents. Calls the stats method to report current metrics upon starting.

```python
def on_start(self):
        if set(['on_start']) & self.o.logEvents:
            self.stats()
            logger.info("starting")
```

--------------------------------

### Install RabbitMQ Server and Configure Users

Source: https://metpx.github.io/sarracenia/How2Guides/Admin_Rabbit_Addendum.html

Installs the rabbitmq-server package and configures various users (anonymous, feeder, root) with specific permissions. Also demonstrates how to manage the guest user and list existing users.

```bash
# installing package taken on the rabbitmq homepage
# rabbitmq-server version > 3.3.x  required to use ldap for passwords verification only

apt-get install erlang-nox
dpkg -i /tmp/rabbitmq-server_3.3.5-1_all.deb

# create anonymous user
# password ********* provided in potato
#                                          conf write read
rabbitmqctl add_user anonymous *********
rabbitmqctl set_permissions -p / anonymous   "^xpublic|^amq.gen.*$|^cmc.*$"     "^amq.gen.*$|^cmc.*$"    "^xpublic|^amq.gen.*$|^cmc.*$"
rabbitmqctl list_user_permissions anonymous

# create feeder user
# password ********* provided in potato
#                                       conf write read
rabbitmqctl add_user feeder ********
rabbitmqctl set_permissions -p / feeder  ".*"  ".*"  ".*"
rabbitmqctl list_user_permissions feeder

# create administrator user
# password ********* provided in potato

rabbitmqctl add_user root   *********
rabbitmqctl set_user_tags root administrator

# takeaway administrator privileges from guest
rabbitmqctl set_user_tags guest
rabbitmqctl list_user_permissions guest
rabbitmqctl change_password guest *************

# list users
rabbitmqctl list_users


# enabling management web application
# this is important since sr_rabbit uses this management facility/port access
# to retrieve some important info

rabbitmq-plugins enable rabbitmq_management
/etc/init.d/rabbitmq-server restart

```

--------------------------------

### Generating Windows Installer

Source: https://metpx.github.io/sarracenia/_sources/Contribution/Release.rst.txt

Generates a Windows installer using pynsist. This script requires Python 3.10 or higher and NSIS to be installed. The output is logged to 'log.txt'.

```bash
# if not already installed:
#   sudo apt install nsis
#   pip3 install pynsist wheel

git checkout pre-release
./generate-win-installer.sh 3.10.11 2>&1 > log.txt
```

--------------------------------

### Start Services and Monitor Logs

Source: https://metpx.github.io/sarracenia/Tutorials/Setup_a_local_subscriber.html

Initializes the necessary directories, starts the 'sr3' service, and begins logging for the watcher. This step initiates the file watching and reporting process.

```bash
$ mkdir -p /tmp/sarra/{in,out}put
$ sr3 start
$ sr3 log watch/test-watch

```

--------------------------------

### Build and Install Debian Package Locally

Source: https://metpx.github.io/sarracenia/Contribution/Development.html

Builds a Debian package for local installation and testing. This command needs to be re-run after each code change and requires python3-amqp to be installed.

```bash
dpkg -i
```

--------------------------------

### Example sr_report Configuration and Output

Source: https://metpx.github.io/sarracenia/_sources/How2Guides/Admin.rst.txt

This example shows a configuration file for sr_report and its subsequent output when run in the foreground. It demonstrates active downloading from a pump and message reception.

```bash
blacklab% more boulelog.conf

broker amqps://feeder@boule.example.com/
exchange xreport
accept .*

blacklab%

blacklab% sr_report foreground boulelog.conf 
2016-03-28 16:29:53,721 [INFO] sr_report start
2016-03-28 16:29:53,721 [INFO] sr_report run
2016-03-28 16:29:53,722 [INFO] AMQP  broker(boule.example.com) user(feeder) vhost(/)
2016-03-28 16:29:54,484 [INFO] Binding queue q_feeder.sr_report.boulelog.06413933.71328785 with key v02.report.# from exchange xreport on broker amqps://feeder@boule.example.com/
2016-03-28 16:29:55,732 [INFO] Received notice  20160328202955.139 http://boule.example.com/ radar/CAPPI/GIF/XLA/201603282030_XLA_CAPPI_1.5_RAIN.gif 201 blacklab anonymous -0.040751
2016-03-28 16:29:56,393 [INFO] Received notice  20160328202956.212 http://boule.example.com/ radar/CAPPI/GIF/XMB/201603282030_XMB_CAPPI_1.5_RAIN.gif 201 blacklab anonymous -0.159043
2016-03-28 16:29:56,479 [INFO] Received notice  20160328202956.179 http://boule.example.com/ radar/CAPPI/GIF/XLA/201603282030_XLA_CAPPI_1.0_SNOW.gif 201 blacklab anonymous 0.143819
2016-03-28 16:29:56,561 [INFO] Received notice  20160328202956.528 http://boule.example.com/ radar/CAPPI/GIF/XMB/201603282030_XMB_CAPPI_1.0_SNOW.gif 201 blacklab anonymous -0.119164
2016-03-28 16:29:57,557 [INFO] Received notice  20160328202957.405 http://boule.example.com/ bulletins/alphanumeric/20160328/SN/CWVR/20/SNVD17_CWVR_282000___01910 201 blacklab anonymous -0.161522
2016-03-28 16:29:57,642 [INFO] Received notice  20160328202957.406 http://boule.example.com/ bulletins/alphanumeric/20160328/SN/CWVR/20/SNVD17_CWVR_282000___01911 201 blacklab anonymous -0.089808
2016-03-28 16:29:57,729 [INFO] Received notice  20160328202957.408 http://boule.example.com/ bulletins/alphanumeric/20160328/SN/CWVR/20/SNVD17_CWVR_282000___01912 201 blacklab anonymous -0.043441
2016-03-28 16:29:58,723 [INFO] Received notice  20160328202958.471 http://boule.example.com/ radar/CAPPI/GIF/WKR/201603282030_WKR_CAPPI_1.5_RAIN.gif 201 blacklab anonymous -0.131236
2016-03-28 16:29:59,400 [INFO] signal stop
2016-03-28 16:29:59,400 [INFO] sr_report stop
blacklab%
```

--------------------------------

### Enable and Start User Systemd Service

Source: https://metpx.github.io/sarracenia/How2Guides/Admin.html

Commands to enable and start the Metpx user service for systemd-based systems. Ensure the user service is enabled to start on every login.

```bash
systemctl --user enable metpx-sr3
systemctl --user start metpx-sr3
```

--------------------------------

### Install RabbitMQ and Configure

Source: https://metpx.github.io/sarracenia/Contribution/Development.html

Installs RabbitMQ server, enables management plugins, and configures broker, credentials, and admin settings. This is a prerequisite for running flow tests.

```bash
sudo apt-get install rabbitmq-server
sudo rabbitmq-plugins enable rabbitmq_management

mkdir ~/.config/sarra
cat > ~/.config/sarra/default.conf << EOF
declare env FLOWBROKER=localhost
declare env MQP=amqp
declare env SFTPUSER=${USER}
declare env TESTDOCROOT=${HOME}/sarra_devdocroot
declare env SR_CONFIG_EXAMPLES=${HOME}/git/sarracenia/sarra/examples
EOF

RABBITMQ_PASS=S0M3R4nD0MP4sS
cat > ~/.config/sarra/credentials.conf << EOF
amqp://bunnymaster:${RABBITMQ_PASS}@localhost/
amqp://tsource:${RABBITMQ_PASS}@localhost/
amqp://tsub:${RABBITMQ_PASS}@localhost/
amqp://tfeed:${RABBITMQ_PASS}@localhost/
amqp://anonymous:${RABBITMQ_PASS}@localhost/
amqps://anonymous:anonymous@hpfx.collab.science.gc.ca
amqps://anonymous:anonymous@hpfx1.collab.science.gc.ca
amqps://anonymous:anonymous@hpfx2.collab.science.gc.ca
amqps://anonymous:anonymous@dd.weather.gc.ca
amqps://anonymous:anonymous@dd1.weather.gc.ca
amqps://anonymous:anonymous@dd2.weather.gc.ca
ftp://anonymous:anonymous@localhost:2121/
EOF

cat > ~/.config/sarra/admin.conf << EOF
cluster localhost
admin amqp://bunnymaster@localhost/
feeder amqp://tfeed@localhost/
declare source tsource
declare subscriber tsub
declare subscriber anonymous
EOF

sudo rabbitmqctl delete_user guest

sudo rabbitmqctl add_user bunnymaster ${RABBITMQ_PASS}
sudo rabbitmqctl set_permissions bunnymaster ".*" ".*" ".*"
sudo rabbitmqctl set_user_tags bunnymaster administrator

sudo systemctl restart rabbitmq-server
cd /usr/local/bin
sudo mv rabbitmqadmin rabbitmqadmin.1
sudo wget http://localhost:15672/cli/rabbitmqadmin
sudo chmod 755 rabbitmqadmin

sr3 --users declare

```

--------------------------------

### Install Sarracenia Debian Package

Source: https://metpx.github.io/sarracenia/Contribution/Development.html

Installs the Sarracenia package as a Debian package from repositories. This method requires installing python3-amqp separately and is typically used by end-users, not for package development.

```bash
apt install metpx-sr3
```

--------------------------------

### on_start Callback

Source: https://metpx.github.io/sarracenia/Reference/flowcb.html

This callback is executed after a connection is established with the broker and before any message transfer occurs. It is used for initial setup.

```python
def on_start(self) -> None:
```
After the connection is established with the broker and things are instantiated, but
before any message transfer occurs.

```
```

--------------------------------

### Declare Environment Variable for Config Examples

Source: https://metpx.github.io/sarracenia/Contribution/man_page_template.html

This snippet shows how to declare an environment variable to specify the location of configuration examples. This is useful for users who want to copy and customize example configurations.

```bash
declare env SR_CONFIG_EXAMPLES=/usr/lib/python3/dist-packages/sarra/examples
```

--------------------------------

### v2 Plugin Configuration Example

Source: https://metpx.github.io/sarracenia/_sources/How2Guides/Plugins_v2ToSr3.rst.txt

This is an example of a v2 configuration for a DESTFNSCRIPT plugin.

```ini
accept .*${HOSTNAME}.*AWCN70_CWUL.*       DESTFNSCRIPT=sender_renamer_add_date.py
```

--------------------------------

### Option Order and Scope Example 1

Source: https://metpx.github.io/sarracenia/_sources/Reference/sr3_options.7.rst.txt

Demonstrates how 'mirror', 'directory', and 'accept' options affect file processing based on their order. In this sequence, 'mirror' is off.

```plaintext
mirror off
directory /data/just_flat_files_here_please
accept .*flatones.*
```

--------------------------------

### Generate Windows Installer Script

Source: https://metpx.github.io/sarracenia/_sources/Contribution/Release.rst.txt

This shell script is used to generate a Windows installer for Sarracenia. It requires a specific Python version and NSIS to be installed. Run this from the root of the Sarracenia git repository.

```shell
sudo apt install nsis
pip3 install pynsist wheel
./generate-win-installer.sh 3.10.11 2>&1 > log.txt
```

--------------------------------

### Build Windows Installer

Source: https://metpx.github.io/sarracenia/Contribution/Release.html

This command initiates the build process for a Windows installer using a shell script. Ensure NSIS and pynsist are installed. The Python version must match the embedded version.

```shell
sudo apt install nsis
pip3 install pynsist wheel
./generate-win-installer.sh 3.10.11 2>&1 > log.txt

```

--------------------------------

### Example MQTT Subscription Topic Construction

Source: https://metpx.github.io/sarracenia/_sources/Reference/sr3_options.7.rst.txt

Demonstrates how sarracenia constructs MQTT subscription topics using queueName, exchange, topicPrefix, and subtopic. This example shows the default behavior.

```text
broker mqtt://user@broker
queueName q_user_Doreen
subtopic #

resulting subscription topic:   $share/q_user_Doreen/xpublic/v03/#
```