{"id":28302472,"url":"https://github.com/giljr/how_to_storing_fiscal_data","last_synced_at":"2025-07-26T19:10:30.161Z","repository":{"id":288491493,"uuid":"968290590","full_name":"giljr/How_To_Storing_Fiscal_Data","owner":"giljr","description":"You now have a production-ready Rails 8 app capable of parsing and storing fiscal data from RCAD-formatted files.","archived":false,"fork":false,"pushed_at":"2025-05-28T11:37:24.000Z","size":590,"stargazers_count":0,"open_issues_count":7,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-31T05:37:33.478Z","etag":null,"topics":["batch-processing","fileutils","fiscalization","moving-processed-file","rails8","rcad","transactions","upload-file"],"latest_commit_sha":null,"homepage":"https://medium.com/jungletronics/building-a-robust-rcad-file-upload-service-with-rails-8-b39bc084c1e3","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/giljr.png","metadata":{"files":{"readme":"README.md","changelog":"history.txt","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-17T20:31:35.000Z","updated_at":"2025-04-18T22:48:47.000Z","dependencies_parsed_at":"2025-04-18T11:07:20.689Z","dependency_job_id":"0813bbe2-c8d7-475e-b812-da04f680e00a","html_url":"https://github.com/giljr/How_To_Storing_Fiscal_Data","commit_stats":null,"previous_names":["giljr/how_to_storing_fiscal_data"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/giljr/How_To_Storing_Fiscal_Data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giljr%2FHow_To_Storing_Fiscal_Data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giljr%2FHow_To_Storing_Fiscal_Data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giljr%2FHow_To_Storing_Fiscal_Data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giljr%2FHow_To_Storing_Fiscal_Data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/giljr","download_url":"https://codeload.github.com/giljr/How_To_Storing_Fiscal_Data/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giljr%2FHow_To_Storing_Fiscal_Data/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267213525,"owners_count":24053912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-26T02:00:08.937Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch-processing","fileutils","fiscalization","moving-processed-file","rails8","rcad","transactions","upload-file"],"created_at":"2025-05-23T21:11:35.146Z","updated_at":"2025-07-26T19:10:30.156Z","avatar_url":"https://github.com/giljr.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🚀 Building a Robust RCAD File Upload Service with Rails 8\n\n### _From Project Setup to Service Execution — A Step-by-Step Guide Using Ruby 3.3+, Transaction, FileUtils and Clean Architecture_\n\n---\n\n## 📘 About the Project\n\nRCAD is a fiscal data initiative by Brazil's **CONFAZ** (National Council for Fiscal Policy), enabling structured communication across all 27 state tax departments. This project simulates a **real-world use case**, where you'll build a robust upload and parsing system using a clean, modular Rails 8 architecture.\n\nThough the data used here is **fictitious**, the structure is suitable for production with minimal adjustments.\n\n---\n\n## 🎯 Objective\n\nCreate a Rails 8 app with:\n\n- ✅ Ruby 3.3+ support\n- ✅ Batch processing with `activerecord-import`\n- ✅ Transaction safety and error rollback\n- ✅ Modular parsing using `.detalhe(row)`\n- ✅ Directory structure for organized file management\n- ✅ API endpoint for triggering the service\n- ✅ Ready for logging, testing, and extension\n\n---\n\n## 💡 Highlights\n\n- 🔄 Batch import with `batch_size: 10000` for performance\n- 🔒 Safe transactions using `ActiveRecord::Base.transaction`\n- 🧱 Clean separation of logic using methods like `cliente(row)`\n- 📁 Files moved after processing with `FileUtils.mv`\n\n---\n\n## 🚀 Let's Get Started!\n\n\u003e ⚠ If you get stuck, feel free to check the [GitHub repo](#).\n\n---\n\n### 1. Project Setup 🛠\n\n```bash\nrails new rcad_app --api\ncd rcad_app\n```\n### 2. Add Required Gems 🧩\n\nIn your Gemfile:\n```ruby\ngem \"activerecord-import\"\ngem \"tty-logger\"\ngem \"tty-spinner\"\n```\nThen run:\n\n    bundle install\n\n### 3. Generate Models 🗃️\n```ruby\nrails g model MeioDeCaptura tipo_tecn:integer term_prop:integer marca:string mestre:references\nrails g model IpParceira cnpj:string nome:string nome_resp:string fone:string email:string mestre:references\nrails g model Adquirencia cnpj_adqui:string mestre:references\nrails g model UfDestinataria uf:string qtd:integer mestre:references\nrails g model Autorizacao tp_autoriz:integer cnpj:string tp_transac:integer dt_ini_aut:date dt_fim_aut:date mestre:references\nrails g model ContadorRegistro qtd_b:integer qtd_c:integer qtd_d:integer qtd_e:integer qtd_f:integer qtd_g:integer qtd_h:integer mestre:references\n```\nAdjust model attributes as needed based on the .detalhe(row) parsing logic.\n\n### 4. Configure File Paths 🔧\n\nIn `config/application.rb`:\n```ruby\nmodule RcadApp\n  class Application \u003c Rails::Application\n    config.load_defaults 8.0\n\n    config.rcad_path = {\n      original: Rails.root.join('rcad_files/original'),\n      processados: Rails.root.join('rcad_files/processados'),\n      erros: Rails.root.join('rcad_files/erros')\n    }\n  end\nend\n```\n### 5. Create Directories\n\n    mkdir -p rcad_files/{original,processados,erros}\n\nYou can now place .TXT files in rcad_files/original.\n### 6. Create the RCAD Service 🧠\n\nFile: app/services/cargas/rcad_carga.rb\n\n    📌 See full implementation above — includes collection setup, transaction handling, row dispatching, and directory movement.\n\n### 7. Implement .detalhe(row) in Models 🧱\n\nExample for Cliente:\n```ruby\nclass Cliente \u003c ApplicationRecord\n  belongs_to :mestre\n\n  def detalhe(row)\n    linha = row.split('|')\n    self.cnpj              = linha[2]\n    self.cpf               = linha[3]\n    self.dt_credenciamento = parse_data(linha[4])\n  end\n\n  private\n\n  def parse_data(str)\n    Date.strptime(str, '%Y%m%d') rescue nil\n  end\nend\n```\nRepeat for other models based on their layout in the RCAD spec.\n### 8. Define Custom Inflection Rules\n\nIn config/initializers/inflections.rb:\n```ruby\nActiveSupport::Inflector.inflections(:en) do |inflect|\n  inflect.irregular 'adquirencia', 'adquirencias'\n  inflect.irregular 'uf_destinataria', 'uf_destinatarias'\n  inflect.irregular 'autorizacao', 'autorizacoes'\nend\n```\nThis ensures proper pluralization when generating models or controllers.\n### 9. Run Migrations 🔃\n\n    rails db:create db:migrate\n\n### 10. Add API Endpoint for File Upload 🧪\nGenerate controller:\n```ruby\nrails g controller api/v1/rcad --skip-template-engine --no-assets --api\n\nAdd action:\n\nmodule Api\n  module V1\n    class RcadController \u003c ApplicationController\n      def processar\n        result = Cargas::RcadCarga.new\n        render json: { message: 'RCAD file processed successfully' }, status: :ok\n      rescue =\u003e e\n        render json: { error: e.message }, status: :unprocessable_entity\n      end\n    end\n  end\nend\n```\nAdd route:\n\n# config/routes.rb\n```ruby\nRails.application.routes.draw do\n  namespace :api do\n    namespace :v1 do\n      post 'rcad/processar', to: 'rcad#processar'\n    end\n  end\nend\n```\n### 11. Test It via Postman or Curl 💬\n\nStart the server:\n\n    rails s\n\nThen run:\n\n    curl -X POST http://localhost:3000/api/v1/rcad/processar\n\nYou should receive:\n\n{\"message\":\"RCAD file processed successfully\"}\n\n### 12. Manual Test via Console 🧪\n\n    rails console\nThen\n    \n    Cargas::RcadCarga.new\n\n## 📈 Improvements Inspired by External Review\n\nSee detailed [Suggestions for Improvement](./IMPROVEMENT.md), based on a review and best practices analysis.\n\n✅ Done!\n\nYou now have a production-ready Rails 8 app capable of parsing and storing fiscal data from RCAD-formatted files.\n## Acknowledgements\n\n - [Jeovan Farias](https://www.linkedin.com/in/jeovan-f-6283b8145/)\n\n\n## Authors\n\n- [Jaythree](https://www.linkedin.com/in/giljrx/)\n\n\n## License\n\n[MIT](https://choosealicense.com/licenses/mit/)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiljr%2Fhow_to_storing_fiscal_data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgiljr%2Fhow_to_storing_fiscal_data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiljr%2Fhow_to_storing_fiscal_data/lists"}