SciXTEMPLATE.TEMPLATE package

Subpackages

SciXTEMPLATE.TEMPLATE.base package
- Module contents

Submodules

SciXTEMPLATE.TEMPLATE.db module

SciXTEMPLATE.TEMPLATE.db.get_TEMPLATE_record(session, record_id): Return record with UUID: record_id

SciXTEMPLATE.TEMPLATE.db.get_job_status_by_job_hash(cls, job_hashes, only_status=None): Return all updates with job_hash

SciXTEMPLATE.TEMPLATE.db.update_job_status(cls, job_hash, status=None): Update status for job previously written to db

SciXTEMPLATE.TEMPLATE.db.write_TEMPLATE_record(cls, record_id, date, s3_key, checksum, source): Write harvested record to db.

SciXTEMPLATE.TEMPLATE.db.write_job_status(cls, job_request, only_status=None): Write new status for job to db

SciXTEMPLATE.TEMPLATE.db.write_status_redis(redis_instance, status)

SciXTEMPLATE.TEMPLATE.models module

class SciXTEMPLATE.TEMPLATE.models.Source(value)

Bases: Enum

An enumeration.

SYMBOL1 = 1

SYMBOL2 = 2

SYMBOL3 = 3

SYMBOL4 = 4

class SciXTEMPLATE.TEMPLATE.models.Status(value)

Bases: Enum

An enumeration.

Error = 3

Pending = 1

Processing = 2

Success = 4

class SciXTEMPLATE.TEMPLATE.models.TEMPLATE_record(**kwargs)

Bases: Base

ArXiV records table table containing the relevant information for harvested arxiv records.

checksum

date

id

s3_key

source

class SciXTEMPLATE.TEMPLATE.models.gRPC_status(**kwargs)

Bases: Base

gRPC table table containing the given status of every job passed through the gRPC API

id

job_hash

job_request

status

timestamp

SciXTEMPLATE.TEMPLATE.template module

class SciXTEMPLATE.TEMPLATE.template.TEMPLATE_APP(proj_home)

Bases: object

session_scope(): Provide a transactional scope for postgres.

template_consumer(consumer, producer): Ingests a message from the Pipeline input topic and passes it to the consumer task

template_task(msg, producer)

input: msg: The consumed msg from the Pipeline input topic producer: The relevant Pipeline output producer

The main consumer task for the Pipeline This task will take any consumed messages and pass them to the relevant subprocesses as well as updating postgres and redis.

SciXTEMPLATE.TEMPLATE.template.init_pipeline(proj_home)

input: proj_home: The home directory for the Pipeline

Initializes the relevant python methods app: The main application class schema_client: The Kafka Schema Registry schema: The input schema the pipeline uses consumer: The kafka consumer for the pipeline producer: The kafka producer for the pipeline