{"id":24775829,"url":"https://github.com/sul-dlss-deprecated/lyberservices-scripts","last_synced_at":"2025-10-04T18:52:51.426Z","repository":{"id":97826403,"uuid":"207914648","full_name":"sul-dlss-deprecated/lyberservices-scripts","owner":"sul-dlss-deprecated","description":"Scripts for object processing outside of our apps (Used to be pre-assembly v3-legacy branch)","archived":false,"fork":false,"pushed_at":"2020-07-13T14:44:54.000Z","size":1429,"stargazers_count":0,"open_issues_count":5,"forks_count":0,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-08-24T18:33:49.662Z","etag":null,"topics":["infrastructure","ruby","scripts"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sul-dlss-deprecated.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-09-11T22:03:38.000Z","updated_at":"2020-07-14T16:50:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"d8977ec7-6f0f-4563-89c4-4cde60a9e348","html_url":"https://github.com/sul-dlss-deprecated/lyberservices-scripts","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sul-dlss-deprecated/lyberservices-scripts","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sul-dlss-deprecated%2Flyberservices-scripts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sul-dlss-deprecated%2Flyberservices-scripts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sul-dlss-deprecated%2Flyberservices-scripts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sul-dlss-deprecated%2Flyberservices-scripts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sul-dlss-deprecated","download_url":"https://codeload.github.com/sul-dlss-deprecated/lyberservices-scripts/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sul-dlss-deprecated%2Flyberservices-scripts/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278358489,"owners_count":25973949,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["infrastructure","ruby","scripts"],"created_at":"2025-01-29T06:55:41.011Z","updated_at":"2025-10-04T18:52:51.352Z","avatar_url":"https://github.com/sul-dlss-deprecated.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lyberservices Scripts\n\n[![Build Status](https://travis-ci.org/sul-dlss/lyberservices-scripts.svg?branch=master)](https://travis-ci.org/sul-dlss/lyberservices-scripts)\n[![Coverage Status](https://coveralls.io/repos/github/sul-dlss/lyberservices-scripts/badge.svg?branch=master)](https://coveralls.io/github/sul-dlss/lyberservices-scripts?branch=master)\n\nA collection of Ruby scripts that may need access to\n- objects before pre-assemble (thumper, smpl drives)\n- dor workspaces (SDR objects before accessioning)\n- stacks\n- web-archiving-stacks\n- preservation storage roots\n\nThese scripts are used by a small number of power users in the PSM group (Peter, for one.)\n\nNote that we hope to retire this repo:\n- if you are writing a new script, please surface it in #dlss-infrastructure channel to see if there is a different way to get the desired result, without adding to our maintenance burden.\n\nThis code was in the sul-dlss/pre-assembly git repo before pre-assembly became a Rails app.\nAfter the Rails version of pre-assembly, this code was in the v3-legacy branch of the sul-dlss/pre-assembly git repo.\n\n## Deployment\n\nRegular capistrano deployment:\n\n```bash\ncap stage deploy # for lyberservices-test\ncap prod deploy # for lyberservices-prod\n```\n\n## Running\n\n0.  If you can, run your job using the pre-assembly web app: https://sul-preassembly-prod.stanford.edu/ instead of using these scripts.\n\n1.  Gather information about your project, including:\n    *   The location of the materials.  You will need read access to this\n        location from the servers you will be accessioning in (e.g. test and\n        production).\n    *   Whether the objects are already registered or not.\n    *   The location of any descriptive metadata.\n    *   Whether you will be flattening the folder structure of each object\n        when accessioning (e.g. discarding any folder structure provided to\n        you in each object).\n    *   The DRUID of the project's APO.\n    *   The DRUID of the set object you will be associating your objects with\n        (if any).\n    *   If your objects are not yet registered and you have a manifest file in\n        CSV format, make sure you have columns for sourceid, filename, and\n        label.  See config/projects/manifest_template/TEMPLATE_manifest.csv\n        for an example manifest.  See the \"manifest\" section below for more\n        information.\n    *   If you are using a manifest file in CSV format and want to create\n        descriptive metadata, create a MODs XML template.  See the\n        \"descriptive metadata\" section below for more details.\n\n\n2.  Create a project-configuration YAML file using the data you gathered\n    above. Store this file in a location where it can be accessed by the\n    server (test or production). You should create a YAML file for each\n    environment specifying the parameters as appropriate. Use the convention\n    of `projectname_environment.yaml`, e.g. `revs_test.yaml`. If you have\n    multiple collections to associate your objects with, you will need to run\n    in multiple batches with multiple YAML files. You can add your collection\n    name to the end of each YAML filename to keep track (e.g.\n    `revs_test_craig.yaml`)\n\n    The YAML file can be stored anywhere that is accessible to the server you\n    are running the code on. However, for simplicity, we recommend you store\n    the YAML at the root of your bundle directory, or create a new project\n    folder, place your YAML file into it and then place your bundle directory\n    into your new project folder. ***PLEASE DO NOT PLACE YOUR YAML FILE INTO THE\n    lyberservices-scripts DIRECTORY ITSELF ANYWHERE ON THE SERVER. IT WILL BECOME HARD\n    TO FIND AND BE SUBJECT TO DELETION WHEN NEW CODE IS DEPLOYED.***\n\n    Example:\n\n    *   Your content is on `/thumpers/dpgthumper-staing/Hummel`\n    *   Create a YAML file at\n        `/thumpers/dpgthumper-staging/Hummel/hummel_test.yaml`\n    *   Move your content (if you can) into\n        `/thumpers/dpgthumper-staging/Hummel/content`\n\n\n    If you cannot move your content, be sure your YAML bundle discovery glob\n    and/or regex are specific enough to correctly ignore your YAML file during\n    discovery. Or, alternatively, place your YAML file in a location other\n    than the bundle.\n\n    *   See [`TEMPLATE.yaml`](config/projects/TEMPLATE.yaml) for a fully documented example of a\n        configuration file.\n    *   See [`manifest_noreg_example.yaml`](config/projects/manifest_noreg_example.yaml) for a specific example\n        using a manifest.\n    *   See [`reg_example.yaml`](config/projects/reg_example.yaml) for a specific example using a\n        file system crawl.\n\n3.  Check the permissions on the bundle directory, iteratively. You need read\n    permissions on all the bundle directory folders and files. You need to\n    have write permissions in the location you plan to write the log file too\n    (often this cannot be the thumper drives since it is mounted as\n    read-only).\n\n4.  You may benefit from running some objects in a local or test environment.\n    If your objects are already registered, this may require pre-registering a\n    sample set in test as well as production using the same DRUIDs that are\n    identified with your content. You may also have to move a small batch of\n    test content to a location that is visible to sul-lyberservices-test.\n    Since the thumper drives are not mounted on the test server, you can use\n    the `/dor/content` mount on test for this purpose.\n\n5.  Make sure you have an APO for your object, and that the\n    administrativeMetadata data stream has the `\u003cassemblyWF\u003e` defined in it.\n    If it does not, go to https://consul.stanford.edu/display/APO/Home and\n    find the \"Current FoXML APO template\" link at the bottom of the page.\n    Download and open the template, find the `\u003cassembly\u003e` node and copy it. Go\n    to Fedora admin for each relevant environment (test/production) and this\n    node to the administrativeMetadata stream. If you don't have this workflow\n    defined in your APO, then the assembly robots will never operate and\n    accessioning will not operate. This APO should be defined using the same\n    DRUID in test and production if you intend to run in both locations.\n\n6.  You can perform a dry discovery run to test your YAML configuration. This\n    run will enumerate the discovered objects, tell you how many files were\n    discovered in each object, check for filename uniqueness in each object,\n    and confirm objects are registered with an APO (for projects where objects\n    are pre-registered). This dry run is particularly important if you are\n    flattening each object's folder structure during pre-assemble (e.g. each\n    object has images in a '00' and '05' directory, but you don't want to\n    retain those folders when accessioning), since you will want to check to\n    make sure each file in a given object has unique filenames. For projects\n    that use manifests for object discovery along with checksum files, you can\n    optionally have checksums computed and confirmed. This is really only\n    useful if you are staging content and not accessioning immediately (since\n    the accessioning process will reconfirm checksums).\n\n    First log into sul-lyberservices-test or -prod as needed, and then cd into\n    the lyberservices-scripts directory, e.g.\n\n```\n        ssh scripts@sul-lyberservices-test.stanford.edu\n        cd lyberservices-scripts/current\n        ROBOT_ENVIRONMENT=test bin/discovery_report YAML_FILE\n```\n\n    You will probably want to run this against a specific environment so it\n    can connect to DOR and confirm registration on the appropriate server,\n    e.g:\n\n```\n        ROBOT_ENVIRONMENT=production bin/discovery_report YAML_FILE\n```\n\n    You will see a report containing:\n    *   the total number of objects discovered\n    *   the names of each discovered object along with the number of files\n        which will be discovered in that object\n    *   any entries (directories or files) in the bundle directory which will\n        **not** be discovered based on your configuration.\n    *   the total number and listing of any objects which have duplicate\n        filenames.  You must resolve the duplicate filenames if you intend to\n        flatten the folder structure when accessioning.\n    *   for manifest style projects, the label and source id along with if all\n        source IDs contained in the manifest are unique\n    *   for manifest style projects, a listing of any folders/files present in\n        the bundle directory that are not referenced in the manifest... some\n        will be expected (such as a checksum file), but this will let you see\n        if any expected images/data are missing from the manifest\n    *   for SMPL style projects, a listing of the the number of files found in\n        the content metadata manifest ... which will let you know if you it\n        has correctly found the object in the `smpl_manifest.csv` file -- a 0\n        would mean none were found or listed, which is a problem\n\n\n    If any errors occur, they will be displayed and a total error count is\n    shown at the bottom.\n\n    To send the report to a CSV file for better sorting and viewing in Excel,\n    send the output to a file using normal UNIX syntax, e.g.:\n\n```\n        ROBOT_ENVIRONMENT=production bin/discovery_report YAML_FILE \u003e /full/path/to/report/filename.csv\n```\n\n    When sending output to a CSV, you will not see any terminal output while\n    the report is running.\n\n    Options for discovery report: You can add the following parameters after\n    the YAML_FILE name. Note that adding each option may make the report time\n    consuming, especially for large number of objects. Some options only work\n    for certain styles of projects.\n\n    `confirm_checksums`:\n        for manifest style projects, will compute and confirm checksums\n        against the checksum file if it exists -- useful it you are not\n        accessioning immediately\n    `check_sourceids`:\n        for manifest style projects, will confirm source IDs are globally\n        unique in DOR (sources ids area already checked for local uniqueness\n        in the manifest)\n    `no_check_reg`:\n        for projects where objects are to be registered, DONT'T check if\n        objects are registered and have APOs (assuming they are supposed to be\n        registered already)\n    `show_staged`:\n        will show all files that will be staged (warning: will produce a lot\n        of output if you have lots of objects with lots of files!)\n    `show_smpl_cm`:\n        will show content metadata that will be generated for each SMPL object\n        using the supplied manifest (warning: will produce a lot of XML output\n        if you have lots of objects with lots of files!)\n\n    e.g.\n\n```\n        ROBOT_ENVIRONMENT=production bin/discovery_report YAML_FILE confirm_checksums check_sourceids \u003e report.csv\n```\n\n7.  To run  locally:\n\n```\n# Normal run.  Will restart and crete a new log file, overwriting any existing log file for that project.\nbin/pre-assemble YAML_FILE\n\n# Run in resume mode, which will automatically pick up where left off based on the log file.  Passing the --resume flag overrides the actual value of resume from the YAML config.\nbin/pre-assemble YAML_FILE --resume\n\n# Run in limit mode (default of 200), which will automatically limit the number of items pre-assembled to 200 regardless of what is set in the YAML file.  Useful with resume.\nbin/pre-assemble YAML_FILE --limit --resume\n\n# Run in limit mode (set to 100), which will automatically limit the number of items pre-assembled regardless of what is set in the YAML file.  Useful with resume.\nbin/pre-assemble YAML_FILE --limit=100 --resume\n```\n\n    Again, you can add `ROBOT_ENVIRONMENT=XXXX` to the beginning of the command\n    to run in test, production or other modes as needed.\n\n8.  Running in the production environment:\n\n    *   Navigate to the production box, in the lyberservices-scripts area.\n    *   Set the ROBOT_ENVIRONMENT=production.\n    *   Run pre-assemble with nohup and in the background (\u0026).\n    *   Optionally, include the `--resume` option to override the resume\n        parameter and set to true.\n    *   Optionally, include the `--limit` option to override the limit\n        paramater.  You can specify the limit, or you can let it default to\n        200.\n\n\n    See the example below:\n```\n        ssh scripts@sul-lyberservices-prod.stanford.edu\n        cd /home/scripts/lyberservices-scripts/current\n        ROBOT_ENVIRONMENT=production nohup bin/pre-assemble YAML_FILE \u0026\n```\n\n    If you want to run multiple nohup jobs simultaneously, you can redirect\n    screen output to a different log file:\n\n```\n        ROBOT_ENVIRONMENT=production nohup bin/pre-assemble YAML_FILE \u003e another_nohup_filename.out 2\u003e\u00261\u0026\n```\n\n    Various ways to monitor progress:\n    1.  The workflow grid in Argo, using your project tag to filter.\n    2.  grep pid PROGRESS_LOG          # Using the filename defined in YAML\n        progress_log_file.\n    3.  tail -999f log/production.log  # Detailed logging info for the\n        lyberservices-scripts project itself.\n    4.  tail -999f nohup.out           # Errors, etc from unix output (or\n        \"another_nohup_filename.out\" in the example above)\n\n\n    Be sure to keep your progress log file somewhere useful and be aware if\n    you restart pre-assemble without using the `--resume` switch, it will be\n    overwritten. You will need the progress log for cleanup and restarting.\n\n9.  Running in batch mode, automatically splitting a large run in groups of\n    smaller jobs, using limits and resume:\n\n```\n    bin/batch_run YAML_CONFIG [LIMIT]\n```\n\nThis will run pre-assemble multiple times sequentially, using resume and\nlimits, allowing the process to end and restart each time. This is useful to\nprevent memory errors on the server when running large jobs.  It will\nautomatically compute the number of items remaining to be run, split the job\nup into the number of jobs needed based on the limit you request, and then\nsequentially run them. You will most likely want to run this in nohup mode,\nsending the output to a file.  If no limit is specified, a default of 200 is\nused.\n\n    e.g.\n```\n    nohup ROBOT_ENVIRONMENT=production bin/batch_run /dor/staging/Revs/mail_box4.yaml \u003e /dor/preassembly/revs/mailander_box4_batch.out 2\u003e\u00261\u0026\n    # will run ALL of the incomplete items in that YAML file in the background, but in groups of the default size of 200, sending output for all runs to the mailander_box4_batch.out file\n\n    nohup ROBOT_ENVIRONMENT=production bin/batch_run /dor/staging/Revs/mail_box4.yaml 100 \u003e /dor/preassembly/revs/mailander_box4_batch.out 2\u003e\u00261\u0026\n    # will run ALL of the incomplete items in that YAML file in the background, but in groups of 100, sending output for all runs to the mailander_box4_batch.out file\n```\n\n## Expert Cheatsheet\n\nHere's a quick summary of the basic execution steps:\n\n    # Goto VM and select environment\n    % ssh scripts@lyberservices-prod\n    % export ROBOT_ENVIRONMENT=production\n\n    # Goto project directory and copy YAML and related files\n    % cd MyProject\n    % YAML=MyProject_$ROBOT_ENVIRONMENT.yaml\n    % BUNDLEDIR=`grep bundle_dir: $YAML | cut -d \\' -f2`\n    % cp $YAML $BUNDLEDIR\n    # optionally copy other required files to $BUNDLEDIR\n\n    # Execute the pre-assemble\n    % cd $HOME/lyberservices-scripts/current\n    % bin/discovery_report $BUNDLEDIR/$YAML\n    % bin/pre-assemble $BUNDLEDIR/$YAML\n\n# Notes\n\nThe assembly robots will automatically create jp2 derivates from any TIFFs,\nJP2s, or JPEGs. If you are working on a legacy project that has JP2s already\nthat were generated from source TIFFs, you should **not** stage those files\nduring pre-assemble, or else you will end up with two JP2s for each TIFF. You\ncan do this by using a regex to exclude .JP2 files or by only staging certain\nsubfolders. If you do stage the JP2 files and they have the same filename as\nthe TIFF (but with a different extension) they will be kept as is (i.e. they\nwill NOT have JP2s re-generated from the source TIFFs). If you do stage the\nJP2 files and they have a different basename than the TIFFs, they WILL be\nre-generated, and you will end up with two copies, in two different resources.\n\n## Setting up code for local development\n\n    # Clone project.\n    git clone git@github.com:sul-dlss/lyberservices-scripts.git\n    cd lyberservices-scripts\n\n    Copy the default configs and use them for local and test.\n\n      cp config/environments/test.example.rb config/environments/test.rb\n      cp config/environments/test.example.rb config/environments/development.rb\n\n    You will need to have a certificate to access DOR-test in order to run integration tests from your laptop.  The certificates are placed\n    in your laptop's \"config/certs\" folder and are referenced in the \"config/environments/test.rb\" file.  Talk to DevOps to get a certificate.\n\n    # Get needed gems.\n    bundle install\n\n    # Confirm that it's working by running the tests as described below.\n\n    To make your life easier, it's easiest to put this in your bash profile so you don't need to identify each time you run a command on your laptop during development:\n\n      export ROBOT_ENVIRONMENT=local\n\n## Running tests\n\n### Prerequisites\n\nYou need exiftool on your system in order to successfully run all of the\ntests.\n\n    RHEL: (RPM to install comming soon)\n    Download latest version from:  http://www.sno.phy.queensu.ca/~phil/exiftool\n\n        tar -xf Image-ExifTool-#.##.tar.gz\n        cd Image-ExifTool-#.##\n        perl Makefile.PL\n        make test\n        sudo make install\n\n    If you are a mac user, use homebrew to install it (if you don't have homebrew installed, you really should: http://brew.sh/):\n        brew install exiftool\n\nTo run all tests, use the command below.  Note that to run integration tests,\nyou will need to be connected to the VPN (even if on campus).\n\n    bin/run_all_tests\n\nTo run the unit tests (fast) and the integration tests (slower) separately,\nuse the commands below.  As noted above, integration tests require VPN.\n\n    bundle exec rspec spec        # no DOR access required, no VPN\n    bundle exec rspec integration # DOR access required, VPN\n\n## Environments\n\nUse the ROBOT_ENVIRONMENT=xxxxx in front of commands to run in a specific\nenvironment.  Current available environments are:\n\n- local:   your laptop\n- development:   development servers\n- test:   test servers\n- production:   production servers\n\nThe server environments define which instance of DOR is connected to, as well\nas the workflow and other services. If you run in the incorrect environment,\nyou will find your objects registered in unexpected places, or you may run\ninto errors when objects you believe should be registered are not.\n\n## Screen Command\n\nIf screen is installed on the server you are using (currently not in\nproduction), another possibility instead of running nohup is to run using the\n\"screen\" command.  (NOTE: currently, screen is not available in production).\n\nStart a new screen by typing:\n\n    $  screen\n\nYou can then start pre-assemble without nohup, just like you would locally:\n\n    $  ROBOT_ENVIRONMENT=production bin/pre-assemble YAML_FILE\n\nYou can then detach from the screen by pressing ctrl-a, ctrl-d and then exit\nfrom the server.\n\nYou can come back to your screen by re-logging into the server, and typing\n\n    $ screen -r\n\nYou can also see a list of available screens by typing\n\n    $ screen -list\n\nFor more info on screen, see http://kb.iu.edu/data/acuy.html\n\n## Troubleshooting\n\n### Seeing an error like this when you try to run pre-assemble or a discovery report?\n\n```\nPsych::SyntaxError: (\u003cunknown\u003e): mapping values are not allowed in this\ncontext at line 37 column 14\n```\n\nIt's probably because your YAML configuration file is malformed. YAML is very\npicky, particularly in tabs, spacing and other formatting trickeries.  You verify\nyour YAML file inside `rails console` or `irb`:\n\n```\nbin/console yaml_config = '/full/path/to/your/config.yaml'\nparams = YAML.load(File.read yaml_config)\n```\n\nIf you get a hash of values back, it parsed correctly.  If you get the\n`Psych::SyntaxError`, it did not.  The line number referenced in the error\nshould help you locate the part of your file that is having issues.  Edit and\ntry loading the YAML again on the console to confirm.\n\n1.  If you don't see all of your objects being discovered or no files are\n    found in discovered objects, check the permissions on the bundle\n    directory. You need read permissions on all the bundle directory folders\n    and files.\n\n2.  Be sure you are running on the correct server in the correct environment.\n    See the \"environment\" section above.\n\n3.  Be sure you have read access to the YAML file you created from the server\n    you are running on.\n\n4.  Be sure you have write access to the location you have specified for your\n    progress log file. When running as scripts on the test and production\n    machines, you will NOT have write access to the thumper drivers. You\n    should store your progress log file elsewhere, such as /dor/preassembly\n\n5.  Check to see if the assembly and accessioning robots are running in the\n    environment you are using. See the \"Starting Robots\" section below. It is\n    not recommended that you start robots in production without consulting the\n    Lyberstructure team.\n\n6.  If you don't see JP2s being created (or recreated) for your content, this\n    is probably due to one of the following problems:\n\n    1.  The content metadata generated by pre-assemble didn't set a resource\n        type or set a resource type other than \"image\" or \"page\". Assembly\n        will only create jp2s for images containing in resources marked as\n        \"image\" or \"page\". Pre-assemble will do this automatically for\n        :simple_image and :simple_book projects, but check the output of the\n        content metadata to be sure.\n\n    2.  The image was not a mimetype of 'image/jpeg' or 'image/tiff'.  Any\n        other mimetype will be ignored.\n\n    3.  Your input image was corrupt or missing a color space profile.  This\n        will usually cause the jp2-create robot to fail and throw an error in\n        that workflow step.\n\n    4.  You had an existing JP2 in the directory that matched a tiff or jpeg.\n        In this case the jp2-create robot will not overwrite any existing\n        files just to be safe.\n\n    5.  You had an existing JP2 in the directory that matched a DPG style\n        filename (e.g. if you had existing tiff called `xx000yy1111_00_01.tif`\n        and a jp2 called `xx000yy1111_05_01.jp2`), you will not get another jp2\n        from that tiff even though there would not be a filename clash, under\n        the principle that it refers to the same image).\n\n\n7.  If 6b or 6c above, it is possible to spot check images to assess the\n    problem:\n\n        $ ssh scripts@lyberservices-prod\n        $ cd ~/lyberservices-scripts/current\n        $ ROBOT_ENVIRONMENT=production bin/console\n        \u003e a=Assembly::Image.new('/full/path/to/image')\n        \u003e a.jp2able? # (if \"false\" then diagnose the problem further)\n        \u003e a.exif['profiledescription'] # (if \"nil\" then it is missing color profile)\n        \u003e a.mimetype # (should be \"image/tiff\" or \"image/jpeg\")\n        \u003e a.width # should give you the image width\n        \u003e a.height # should give you image height\n\n    It is possible to force add color profiles to a single image or all of the\n    images in a given directory:\n```\n        source_img=Assembly::Image.new('/input/path_to_file.tif') # add to a single image\n        source_img.add_exif_profile_description('Adobe RGB 1998')\n```\n    or\n```\n        Assembly::Images.batch_add_exif_profile_description('/full_path_to_tifs','Adobe RGB 1998')    # add to multiple images\n```\n\n8.  If you see incorrect content metadata being generated, note that if\n    should_register = false, the 'Process : Content Type' tag for each\n    existing object will be examined. If a known type is set in this tag, it\n    will be used to create content metadata instead of the default set in\n    [project_style](:content_structure). Check the tag for each object if the\n    style is not matching what you have set in the YAML configuration file.\n    Also note that if `content_md_creation[:style]` is set to 'none', then no\n    content metadata will be generated.\n\n\n## Restarting a job\n\nIf you have failed objects during your pre-assemble, these will either cause\npre-assemble to terminate immediately (if the failure is non-recoverable) or\nit will continue and log the errors. The progress log file you specified in\nyour YAML configuration will contain information about which bundles failed.\nYou can re-start pre-assemble and ask it to re-try the failed objects and\ncontinue with any other objects that it hadn't done yet. To do this, use the\n`--resume` flag when you run pre-assemble:\n\n```\n    ROBOT_ENVIRONMENT=production bin/pre-assemble YAML_FILE --resume\n```\n\n## Post Accessioning Reports\n\nUse [Argo](https://argo.stanford.edu/).\n\nTwo reports are available if you can't use Argo.  If you wish to use these\nreports, both produce the following output, but differ in how they locate\nobjects to report on. The output for both reports is a CSV file in the \"log\"\nfolder of your checked out lyberservices-scripts code. Both will report on up to 50,000\nrows and includes the following columns:\n\n* druid\n* label\n* source_id\n* dc:title\n* published status\n* shelved status\n* PURL url\n* total files in object\n* number of files by file extension\n\n### project tag report\n\nThe first report is called a `\"project_tag_report\"` and includes ALL objects in\nDOR tagged with a specific project tag. This is useful for a global project\noverview and is cumulative (i.e. as more objects are added with that tag, the\nreport will be bigger if run again).\n\n```\nROBOT_ENVIRONMENT=production bin/project_tag_report PROJECT_TAG\n```\n\nwhere `PROJECT_TAG` is the Argo project tag (e.g. \"Revs\"). If your project tag\nhas spaces in it, be sure to use quotes, like this:\n\n```\nROBOT_ENVIRONMENT=production bin/generate_collection_report \"Stanford Oral History Project\"\n```\n\n## Manifests\n\nManifests are a way of indicating which objects you will be accessioning. A\nmanifest file is a CSV, UTF-8 encoded file and works for projects which have\none file per object (container = one file in this case), or projects with many\nfiles per object (container = folder in this case).\n\n**WARNING**: if you export from Microsoft Excel, you may not get a properly\nformatted UTF-8 CSV file. You should open any CSV that has been exported from\nExcel in a text editor and re-save it in proper UTF-8 files (like TextMate on\na Mac).\n\nThere are a few columns required in the manifest, depending on whether `should_register` is `true` or `false`:\n\n- `container`: container name (either filename or folder name) -- **required**\n- `druid`: druid of object (**required** if `should_register` = `false`)\n- `sourceid`: source ID (**required** if `should_register` = `true`)\n- `label`: label (**required** if `should_register` = `true`)\n\nThe druids should include the \"druid:\" prefix (e.g. \"druid:oo000oo0001\" instead of \"oo000oo0001\").\n\nThe first line of the manifest is a header and specifies the column names.\nColumn names should not have spaces and it is easiest if they are all lower\ncase. These columns are used to register objects and indicate which file goes\nwith the object. If the container column specifies a filename, it should be\nrelative to the manifest file itself. You can have additional columns in your\nmanifest which can be used to create descriptive metadata for each object. See\nthe section below for more details on how this works.\n\nThe actual names of the columns above (except for \"druid\") can be set in the\nYAML file.  The druid column **must** be called `\"druid\"`..\n\nSee the sample manifest file [`TEMPLATE_manifest.csv`](config/projects/manifest_template/TEMPLATE_manifest.csv)\n\n## Descriptive Metadata\n\nIf descriptive metadata is supplied in a source known to common accessioning\n(currently MDToolkit or Symphony), then no action is required during\npre-assemble other than ensuring your DRUIDs and/or barcodes match the ones in\nMDToolkit or Symphony.\n\nIf you are supplying a manifest file instead of using object discovery via\nfile system crawling, then you can also create a descriptive metadata MODs\nfile for each object using content supplied in the manifest.  By creating a\ntemplate XML MODS file, placing with your YAML configuration file and ensuring\nit's filename is indicated in your YAML configuration, you can tell\npre-assemble to generate a MODs file per object.  The generated MODs file\nshould be called \"descMetadata.xml\" and will be staged alongside the content.\nThis file is automatically picked up during common accessioning.\n\nThe MODs file is generated by taking the XML template you supply, and filling\nin any `[[field]]` values in the template with the corresponding column from\nthe manifest.\n\nFor example, if your template has\n\n```xml\n\u003cmods\u003e\u003ctitle\u003e[[description]]\u003c/title\u003e\u003c/mods\u003e\n```\n\nand you have a column called \"description\" in your manifest and you have a row\nwith a value of \"picture of me\", you will get that value filled into your\ntemplate for that specific object:\n\n```xml\n\u003cmods\u003e\u003ctitle\u003epicture of me\u003c/title\u003e\u003c/mods\u003e\n```\n\nIn addition, the entire MODs template is passed through an ERB parser,\nallowing you to utilize Ruby code in the template using the standard\n[ERB](http://ruby-doc.org/stdlib-1.9.3/libdoc/erb/rdoc/ERB.html) template `\u003c%\n%\u003e` syntax.  This can be used to perform more complex operations.  If you\nutilize Ruby code, you will have access to a special local variable called\n'manifest_row', which is a hash of values for that row in the manifest, keyed\noff the column names.  For example:\n\n```xml\n\u003cmods\u003e\u003ctitle\u003e\u003c%= manifest_row[:description] %\u003e\u003c/title\u003e\u003c/mods\u003e\n```\n\nwill provide the same output as the previous example.  A full example of a\nMODs template is provided at\nconfig/projects/manifest_template/TEMPLATE_mods.xml\n\nTo use a different character encoding in your ERB template, put the following\nat the top of your *template.xml*:\n\n```xml\n\u003c%#-*- coding: UTF-8 -*-%\u003e\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n```\n\n## Testing Descriptive Metadata Generation\n\nIf you would like to test your MODs template prior to actually accessioning,\nyou can run a \"mods report\", passing in the YAML config file, which references\nyour manifest and MODs template, and a writable output folder location.  The\nreport will then generate a MODs file for each row in your manifest so you can\nexamine the results.  You can limit the number of rows run by temporarily\nmodifying the \"limit_n\" parameter in the YAML file. Note that the output\nfolder MUST exist and must be writable.  Be aware it will become filled with\nMODs files, one per object.  So if you have a large number of rows in your\nmanifest, you will end up with many files in your output directory.\n\nROBOT_ENVIRONMENT=production bundle exec bin/mods_report YAML_CONFIG_FILE\nOUTPUT_DIRECTORY\n\n## Accession of Specific Objects\n\nFor projects with a manifest (e.g. like Revs):\n\n1.  Create a new manifest with only the objects you need accessioned.\n2.  Create a new project config YAML file referencing the new manifest and\n    write to a new progress log file.\n3.  Run pre-assemble script.\n\n\nFor projects that do not use a manifest and which have their objects already\nregistered (e.g. like Rumsey):\n\n1.  Create a new project config YAML file and set the parameter\n    'accession_items' using either the 'only' or\n\n'except' parameter as needed. You can include only specific objects (useful\nwhen you only want to run a few objects) or you can exclude specific objects\n(useful when you want to run most). Also set a different progress log file so\nyou can store the results of your second run separately. See the TEMPLATE.yaml for some examples.\n\n1.  Run pre-assemble.\n\n\n\n### Loading YAML Configuration\n\nIf you are working in the console, and want to read your YAML configuration\nfile (for example, to determine where your progress log file is located), you\ncan use the following methods to load the configuration into a ruby hash:\n\ne.g.\n\n```\n$ ROBOT_ENVIRONMENT=test bin/console\n\nconfig_filename='/thumpers/dpgthumper2-smpl/SC1017_SOHP/sohp_prod_accession.yaml'\nconfig=YAML.load(IO.read(filename))\nprogress_filename=config['progress_log_file']\n```\n\nYou can then use these values in other utility methods as needed.\n\n### Finding Druids\n\nIf you want to find the druids from your progress log file that are either\ncompleted or not completed, you can use a method that will give you an array\nof relevant druids. You can then use this array in 'workflow_status' method\nnoted above or in the other utility methods.\n\ne.g.\n```\n$ ROBOT_ENVIRONMENT=test bin/console\n\ncompleted_druids=PreAssembly::Utils.get_druids_from_log('/dor/preassembly/sohp_accession_log.yaml',true)\nfailed_druids=PreAssembly::Utils.get_druids_from_log('/dor/preassembly/sohp_accession_log.yaml',false)\n\n```\n\nIf you want to find druids by source_id, use the utility method\nPreAssembly::Utils.get_druids_by_sourceid(source_ids=[]) to do this. You can then\nuse the array of druids in the other utility methods.\n\ne.g.\n\n    $ ROBOT_ENVIRONMENT=test bin/console\n\n    source_ids=%w{foo:123 bar:456}\n    druids=PreAssembly::Utils.get_druids_by_sourceid(source_ids)\n\n## Remediation\n\n### Basic Object Remediation\n\nA basic object remediation framework is provided for mass remediating objects.\n\nTo use it, first decide how you will specify the list of druids.  You can\neither manually specify with a CSV (see step 1 below), or use the pre-assemble\nYAML log file if you are remediating a run from pre-assemble.\n\n1.  For running a specific list of druids, create a CSV file containing a list\n    of druids you wish to remediate, one line per druid, with a header column\n    called \"druid\" or \"Druid\" (caps don't matter).\n\nThere can be other columns too -- they will be ignored.  They can be either\nfull druids (with the prefix) or just the PIDs.  Save it somewhere the\nlyberservices-scripts code can read it.\n\ne.g.\ndruid\ndruid:oo000oo0001\ndruid:oo000oo0002\noo000oo0003\n\n1.  Create a ruby file that defines exactly how you need to remediate each\n    object.  The ruby file will define a method that has access to the\n\nfedora object and can perform any actions needed on the object.  The actual\nloading, saving, and versioning of objects is handled by the framework - you\njust need to define the actual logic needed to operate on the object.  The\nfile needs to have a specific format with two specific methods that must be\ndefined, one that determines if remediation will occur at all, and the second\nindicates the type of remediation to perform.  An example file and its format\nis shown in the file 'lib/remediation/remediate_project_example.rb' Don't edit\nthat file - copy it, and edit it somewhere the script can read it.\n\nNote that you have access to 'equivalent-xml'\n(https://github.com/mbklein/equivalent-xml) for comparing xml when deciding if\nyou need to remediate.  You also can use nokogiri. If you use equivalent-xml\nbe sure to require it at the top of your script.\n\n1.  Run with the command below (presumably on the lyberservices-prod server,\n    which has access to Fedora production, although you can also\n\nrun in test or development mode if you can access those environments).  Pass\nin the either the CSV file with DRUIDs or the pre-assemble YAML file and the\nRuby file you generated in steps 1 and 2.  You will get two output files --\nthe first is a CSV file which includes columns indicating if remediation\nsucceeded, a message and a timestamp.  You will also get a .log file.  Each\nfile is named in the same way as the input file with _log.csv and _log.yml\nappended to the filename and is placed in the same location.  This means you\nshould place the input CSV in be a location where the script will have write\naccess to that location (i.e. not on a thumper drive if you are on\nlyberservices-prod).\n\nNote that the _log.yml file is used to ensure that objects are not run through\nremediation twice, so you should keep that file in the same location as the\ninput CSV if you need to resume a large remediation.  In each case, the file\nwill always be appended to.\n\nROBOT_ENVIRONMENT=production bin/remediate INPUT_FILE.CSV[or LOG_FILE.YAML]\nREMEDIATE_LOGIC_FILENAME.rb\n\nFor long running actions, you can run in nohup mode\n\nROBOT_ENVIRONMENT=production nohup bin/remediate INPUT_FILE.CSV[or\nLOG_FILE.YAML] REMEDIATE_LOGIC_FILENAME.rb \u0026\n\nThe result will be some screen output and a detailed log file in .YML format\nin the same location and with the same name as the input CSV file. The input\nCSV file will also be updated with two additional columns - a status\nindicating if remediation succeeded and a message. You can re-run the\nremediation with the same CSV file and it will automatically skip already\ncompleted objects - SO KEEP THE CSV FILE.\n\nROBOT_ENVIRONMENT=production bin/remediate INPUT_FILE.csv\nREMEDIATE_LOGIC_FILENAME.rb [PAUSE_TIME_IN_SECONDS]\n\nFinally, you can specify a pause time in seconds per object. This can be\nuseful for large remediation jobs that are a low priority, which allows them\nto run at a slower pace and thus generating less concurrent load on the system\nwhile other higher priority jobs are running concurrently.\n\n### MODs Remediation\n\nA custom script has already been written to handle the case of updating MODs\ngiven a spreadsheet and a MODs template.  See the file\ndevel/update_mods_metadata.rb for more information.\n\n### Custom Object Remediation\n\nYou can also build a fully customized remediation script that does not require\ninput DRUIDs in a CSV and can pass data to individual druids for more specific\nremediation.  To do this, you will need to implement the logging or resume\ncapability you will need. You will also still implement your own project\nspecific Ruby file (as in step 2 above) but you will also implement your own\nmechanism for generating druids to remediate and for logging their completion.\n\nWithin the lyberservices-scripts codebase, you will have access to the following\nremediation class.   Pass it a druid and your remediation logic Ruby file, and\nit will return you success or failure and a message.  You can also use the\nbuilt in logging methods to record success/failure to help with resuming.  You\ncan also optionally pass any data structure or object you need and have that\navailable in your custom method.\n\n    project_file='my_remediation.rb'\n    pid='druid:oo000oo0001'\n    require project_file\n\n    item=PreAssembly::Remediation::Item.new(pid,optionalDataStructure) # optionalDataStructure is some kind of object, string, hash, etc.\n                                                                       # it will be made available to your \"remediate_logic\" method in\n     \t                                                                # the instance variable @data\n    item.extend(RemediationLogic) # add in our project specific methods\n    item.description='some description that will be added to any versions which are opened' # optional\n    success=item.remediate # returns true or false\n    puts item.message # some more info about what was done\n\nLoop over your PIDs and log results as needed.  You can log to an output CSV\nand/or a YML file if you wish, using the following methods\n\n    item.log_to_progress_file(progress_log_file)  # pass in a fully qualified path to a YML file to append to\n    item.log_to_csv(csv_out)   # pass in a fully qualified path to a CSV file to append to\n\nYou can read in completed druids from a progress_log_file you have previously\ncreated using a class level method if you wish to check if a druid is already\ncompleted.  Each of these calls gives you an array of druids.\n\n    completed_druids=PreAssembly::Remediation::Item.get_druids(progress_log_file)\n    failed_druids=PreAssembly::Remediation::Item.get_druids(progress_log_file,false)\n    done if completed_druids.include?(pid)  # will give you a true or false if your current pid is already done\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsul-dlss-deprecated%2Flyberservices-scripts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsul-dlss-deprecated%2Flyberservices-scripts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsul-dlss-deprecated%2Flyberservices-scripts/lists"}