#956456 ITP: idseq-dag -- Pipeline engine for IDseq (Infectious Disease Sequencing Platform)

#956456#5
Date:
2020-04-11 15:00:35 UTC
From:
To:
* Package name    : idseq-dag
  Version         : 4.2.2
  Upstream Author : IdSeq Team @ Chan Zuckerberg Initiative
  <idseqhelp@chanzuckerberg.com>
* URL             : https://github.com/chanzuckerberg/idseq-dag
* License         : MIT
  Programming Lang: Python
  Description     : Pipeline engine for IDseq (Infectious Disease Sequencing Platform)

idseq_dag is the pipeline execution engine for idseq (see idseq.net). It is a
pipelining system that implements a directed acyclic graph (DAG) where the nodes
(steps) correspond to individual python classes. The graph is defined using JSON.
.
The pipeline would be executed locally with local machine resources. idseq-dag
could be installed inside a docker container and run inside the container.
.
This is package for COVID-19 Hackathon. This package can be maintained
under med-team umbrella.

#956456#10
Date:
2020-04-15 15:22:26 UTC
From:
To:
Hi Emmanuel,

I've seen your ITP of idseq-dag.  You mentioned

   idseq-dag depends on pytz. I will send the ITP for that package.

I admit its a bit confusing but that package is python3-tz in Debian, is
packaged and I've added it to Build-Depends (amongst other changes).

The current packaging is suffering from failures inside the test suite
looking like this:


======================================================================
ERROR: test_step_single (tests.generate_alignment_viz.GenerateAlignmentVizTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/idseq-dag-4.2.3/.pybuild/cpython3_3.8_idseq-dag/build/tests/generate_alignment_viz.py", line 9, in test_step_single
    run_step = IdseqStepSetup.get_step_object(PipelineStepGenerateAlignmentViz, "alignment_viz_out", paired=False)
  File "/build/idseq-dag-4.2.3/.pybuild/cpython3_3.8_idseq-dag/build/tests/idseq_step_setup.py", line 47, in get_step_object
    PipelineFlow.fetch_input_files_from_s3(
  File "/build/idseq-dag-4.2.3/.pybuild/cpython3_3.8_idseq-dag/build/idseq_dag/engine/pipeline_flow.py", line 179, in fetch_input_files_from_s3
    output_file = idseq_dag.util.s3.fetch_from_s3(s3_file, local_dir, allow_s3mi=True)
  File "/build/idseq-dag-4.2.3/.pybuild/cpython3_3.8_idseq-dag/build/idseq_dag/util/s3.py", line 286, in fetch_from_s3
    if is_reference or os.path.abspath(dst).startswith(config["REF_DIR"]):
TypeError: startswith first arg must be str or a tuple of str, not NoneType


The issue is that I had to replace some default dir in a patch[1].
Unfortunately this did not propagated into the config dictionary and
the key "REF_DIR" is not set.  For the moment I gave up here - may
be you want to continue by either fixing my patch or finding a better
way to convince the test suite to write to some place where this is
permitted.

Kind regards

       Andreas.

[1] https://salsa.debian.org/med-team/idseq-dag/-/blob/master/debian/patches/set_default_output_dir_var_tmp.patch

#956456#15
Date:
2020-04-15 15:50:51 UTC
From:
To:
Hi Andreas,

Thanks for the notice that pytz is already on Debian.

And thanks so much for the patch :) I will continue working on that and
trying to
fix the issue.

I will keep you updated

Cheers,
Arias Emmanuel
@eamanu
yaerobi.com