{"id":19509605,"url":"https://github.com/raspi/scrapy-intel-ark","last_synced_at":"2025-10-05T22:59:27.020Z","repository":{"id":138159905,"uuid":"234936941","full_name":"raspi/scrapy-intel-ark","owner":"raspi","description":"Web crawler for Intel ARK (ark.intel.com)","archived":false,"fork":false,"pushed_at":"2021-05-18T01:19:18.000Z","size":55,"stargazers_count":14,"open_issues_count":1,"forks_count":5,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-26T03:44:06.418Z","etag":null,"topics":["hardware","intel","python","scrapy","spider","webcrawler"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raspi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-19T17:22:52.000Z","updated_at":"2024-04-21T20:34:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"06169573-b266-4f44-b8d1-09199587abe0","html_url":"https://github.com/raspi/scrapy-intel-ark","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/raspi/scrapy-intel-ark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fscrapy-intel-ark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fscrapy-intel-ark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fscrapy-intel-ark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fscrapy-intel-ark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raspi","download_url":"https://codeload.github.com/raspi/scrapy-intel-ark/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fscrapy-intel-ark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278532353,"owners_count":26002346,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-05T02:00:06.059Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hardware","intel","python","scrapy","spider","webcrawler"],"created_at":"2024-11-10T23:12:39.075Z","updated_at":"2025-10-05T22:59:27.000Z","avatar_url":"https://github.com/raspi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# scrapy-intel-ark\nWeb crawler for Intel ARK ([ark.intel.com](https://ark.intel.com))\n\n## Requirements\n\n* Python\n* [Scrapy](https://scrapy.org/)\n\n## Notes\n\n* 30 day cache is used in `settings.py`\n* Some product information pages do **not** contain socket information, so they are written to `_unknown/` directory\n\n## Spiders\n\n### CPU specs spider\n\nDownloads all CPU specifications as JSON files.\n\n    scrapy crawl cpuspecs\n    \nEverything is downloaded to `items/cpuspecs` directory. Each CPU is in it's own socket subdirectory. \n\nAfter crawling `_legend.json` is written which has explanations for the fields.\n\n### Single CPU specs spider\n    \n    scrapy crawl onecpuspec -a url=\"https://ark.intel.com/content/www/us/en/ark/products/82764/intel-xeon-processor-e5-1630-v3-10m-cache-3-70-ghz.html\"\n\nSee **CPU specs spider** for info.\n\n### CPU series spider\n\nCrawl only certain CPU series, for example \"2nd Generation Intel® Xeon® Scalable Processors\":\n\n    scrapy crawl series -a url=\"https://ark.intel.com/content/www/us/en/ark/products/series/192283/2nd-generation-intel-xeon-scalable-processors.html\"\n\n\n## `_legend.json`\n\n```json\n{\n  \"Essentials\": {\n    \"ProductGroup\": \"Product Collection\",\n    \"CodeNameText\": \"Code Name\",\n    \"MarketSegment\": \"Vertical Segment\",\n    \"ProcessorNumber\": \"Processor Number\",\n    \"StatusCodeText\": \"Status\",\n    \"BornOnDate\": \"Launch Date\",\n    \"Lithography\": \"Lithography\",\n    \"CertifiedUseConditions\": \"Use Conditions\",\n    \"ExpectedDiscontinuanceDate\": \"Expected Discontinuance\",\n    \"ItemsIncluded\": \"Included Items\",\n    \"WarrantyLengthYears\": \"Warranty Period\"\n  },\n  \"Performance\": {\n    \"CoreCount\": \"# of Cores\",\n    \"ThreadCount\": \"# of Threads\",\n    \"ClockSpeed\": \"Processor Base Frequency\",\n    \"ClockSpeedMax\": \"Max Turbo Frequency\",\n    \"Cache\": \"Cache\",\n    \"Bus\": \"Bus Speed\",\n    \"MaxTDP\": \"TDP\",\n    \"FSBParity\": \"FSB Parity\",\n    \"CoreVoltage\": \"VID Voltage Range\",\n    \"ScenarioDesignPower\": \"Scenario Design Power (SDP)\",\n    \"TurboBoostTech2MaxFreq\": \"Turbo Boost Technology 2.0 Frequency\",\n    \"BusNumPorts\": \"# of QPI Links\",\n    \"ConfigTDPMin\": \"Configurable TDP-down\",\n    \"ConfigTDPMinFrequency\": \"Configurable TDP-down Frequency\",\n    \"BurstFrequency\": \"Burst Frequency\",\n    \"TurboBoostMaxTechMaxFreq\": \"Turbo Boost Max Technology 3.0 Frequency\",\n    \"UltraPathInterconnectLinks\": \"# of UPI Links\",\n    \"ConfigTDPMaxFrequency\": \"Configurable TDP-up Frequency\",\n    \"ConfigTDPMax\": \"Configurable TDP-up\",\n    \"ThermalVelocityBoostFreq\": \"Thermal Velocity Boost Frequency\",\n    \"SingleCoreBaseFrequency\": \"Processor Single Core Base Frequency\",\n    \"SingleCoreTDP\": \"Single Core TDP\"\n  },\n  \"Supplemental Information\": {\n    \"Embedded\": \"Embedded Options Available\",\n    \"null\": \"Datasheet\",\n    \"ProductDescription\": \"Description\",\n    \"ProductBriefUrl\": \"Product Brief\",\n    \"DatasheetUrl\": \"Datasheet\"\n  },\n  \"Memory Specifications\": {\n    \"MaxMem\": \"Max Memory Size (dependent on memory type)\",\n    \"MemoryTypes\": \"Memory Types\",\n    \"NumMemoryChannels\": \"Max # of Memory Channels\",\n    \"MaxMemoryBandwidth\": \"Max Memory Bandwidth\",\n    \"ECCMemory\": \"ECC Memory Supported\",\n    \"PhysicalAddressExtension\": \"Physical Address Extensions\",\n    \"MemoryMaxSpeedMhz\": \"Maximum Memory Speed\",\n    \"OptaneDCPersistentMemoryVersion\": \"Optane DC Persistent Memory Supported\"\n  },\n  \"Processor Graphics\": {\n    \"ProcessorGraphicsModelId\": \"Processor Graphics\",\n    \"GraphicsFreq\": \"Graphics Base Frequency\",\n    \"GraphicsMaxFreq\": \"Graphics Max Dynamic Frequency\",\n    \"QuickSyncVideo\": \"Quick Sync Video\",\n    \"InTru3D\": \"InTru 3D Technology\",\n    \"FDI\": \"Flexible Display Interface ( FDI)\",\n    \"CVTHD\": \"Clear Video HD Technology\",\n    \"NumDisplaysSupported\": \"# of Displays Supported\",\n    \"GraphicsDeviceId\": \"Device ID\",\n    \"GraphicsOutput\": \"Graphics Output\",\n    \"ClearVideoTechnology\": \"Clear Video Technology\",\n    \"LicenseRequired\": \"Macrovision* License Required\",\n    \"GraphicsMaxMem\": \"Graphics Video Max Memory\",\n    \"GraphicsMaxResolutionHDMI\": \"Max Resolution (HDMI 1.4)\",\n    \"GraphicsMaxResolutionDP\": \"Max Resolution (DP)\",\n    \"GraphicsMaxResolutionIFP\": \"Max Resolution (eDP - Integrated Flat Panel)\",\n    \"GraphicsMaxResoluionVGA\": \"Max Resolution (VGA)\",\n    \"GraphicsDirectXSupport\": \"DirectX* Support\",\n    \"GraphicsOpenGLSupport\": \"OpenGL* Support\",\n    \"Graphics4KSupportLevel\": \"4K Support\",\n    \"GraphicsExecutionUnits\": \"Execution Units\",\n    \"GraphicsBurstFrequency\": \"Graphics Burst Frequency\",\n    \"GraphicsMaxRefreshRateHz\": \"Max Refresh Rate\",\n    \"EmbeddedDramMB\": \"eDRAM\"\n  },\n  \"Expansion Options\": {\n    \"PCIExpressRevision\": \"PCI Express Revision\",\n    \"PCIExpressConfigs\": \"PCI Express Configurations\",\n    \"NumPCIExpressPorts\": \"Max # of PCI Express Lanes\",\n    \"ScalableSockets\": \"Scalability\",\n    \"PCISupport\": \"PCI Support\"\n  },\n  \"Package Specifications\": {\n    \"SocketsSupported\": \"Sockets Supported\",\n    \"MaxCPUs\": \"Max CPU Configuration\",\n    \"ThermalSolutionSpecification\": \"Thermal Solution Specification\",\n    \"TCase\": \"TCASE\",\n    \"PackageSize\": \"Package Size\",\n    \"ThermalJunctionRateCode\": \"TJUNCTION\",\n    \"DieSize\": \"Processing Die Size\",\n    \"TransistorCount\": \"# of Processing Die Transistors\",\n    \"OperatingTemperature\": \"Operating Temperature Range\",\n    \"OperatingTemperatureMax\": \"Operating Temperature (Maximum)\",\n    \"OperatingTemperatureMin\": \"Operating Temperature (Minimum)\",\n    \"BracketHeight\": \"Bracket Height\"\n  },\n  \"Advanced Technologies\": {\n    \"TBTVersion\": \"Turbo Boost Technology\",\n    \"VProTechnology\": \"vPro Platform Eligibility\",\n    \"HyperThreading\": \"Hyper-Threading Technology\",\n    \"VTX\": \"Virtualization Technology (VT-x)\",\n    \"VTD\": \"Virtualization Technology for Directed I/O (VT-d)\",\n    \"ExtendedPageTables\": \"VT-x with Extended Page Tables (EPT)\",\n    \"TransactionalSynchronizationExtensionVersion\": \"Transactional Synchronization Extensions\",\n    \"EM64\": \"64\",\n    \"InstructionSet\": \"Instruction Set\",\n    \"InstructionSetExtensions\": \"Instruction Set Extensions\",\n    \"HaltState\": \"Idle States\",\n    \"SpeedstepTechnology\": \"Enhanced SpeedStep Technology\",\n    \"ThermalMonitoring2Indicator\": \"Thermal Monitoring Technologies\",\n    \"IdentityProtectionTechVersion\": \"Identity Protection Technology\",\n    \"MyWiFiTech\": \"My WiFi Technology\",\n    \"WiMAX\": \"4G WiMAX Wireless Technology\",\n    \"DemandBasedSwitching\": \"Demand Based Switching\",\n    \"FastMemoryTechnology\": \"Fast Memory Access\",\n    \"FlexMemoryTechnology\": \"Flex Memory Access\",\n    \"VTI\": \"Virtualization Technology for Itanium (VT-i)\",\n    \"OptaneMemorySupport\": \"Optane Memory Supported\",\n    \"StableImagePlatformProgramVersion\": \"Stable Image Platform Program (SIPP)\",\n    \"SpeedShiftTechVersion\": \"Speed Shift Technology\",\n    \"SmartResponseTechVersion\": \"Smart Response Technology\",\n    \"ThermalVelocityBoostVersion\": \"Thermal Velocity Boost\",\n    \"TurboBoostMaxTechVersion\": \"Turbo Boost Max Technology 3.0\",\n    \"RapidStorageTech\": \"Rapid Storage Technology\",\n    \"SecureBootTechVersion\": \"Secure Boot\",\n    \"HDAudioTechnology\": \"HD Audio Technology\",\n    \"SmartConnectVersion\": \"Smart Connect Technology\",\n    \"AMTVersion\": \"ME Firmware Version\",\n    \"MatrixStorageTechnology\": \"Matrix Storage Technology\",\n    \"AVX512FusedMultiplyAddUnits\": \"# of AVX-512 FMA Units\",\n    \"DeepLearningBoostVersion\": \"Deep Learning Boost ( DL Boost)\",\n    \"ResourceDirectorTechVersion\": \"Resource Director Technology ( RDT)\",\n    \"QuickAssistTechnology\": \"Integrated Quick Assist Technology\",\n    \"VolumeManagementDeviceVersion\": \"Volume Management Device (VMD)\",\n    \"SmartIdleTechVersion\": \"Smart Idle Technology\",\n    \"QuickResumeTechnology\": \"Quick Resume Technology\",\n    \"QuietSystemTechnology\": \"Quiet System Technology\",\n    \"AC97Technology\": \"AC97 Technology\",\n    \"IOAccelerationTechnology\": \"I/O Acceleration Technology\",\n    \"InstructionReplayTechVersion\": \"Instruction Replay Technology\",\n    \"SstPerformanceProfileVersion\": \"Speed Select Technology - Performance Profile\",\n    \"SstBaseFrequencyOptionVersion\": \"Speed Select Technology - Base Frequency\",\n    \"OmniPathArchitectureVersion\": \"Integrated Omni-Path Architecture ( OPA)\"\n  },\n  \"Security \u0026 Reliability\": {\n    \"AESTech\": \"AES New Instructions\",\n    \"SecureKeyTechVersion\": \"Secure Key\",\n    \"TXT\": \"Trusted Execution Technology\",\n    \"ExecuteDisable\": \"Execute Disable Bit\",\n    \"AntiTheftTechnology\": \"Anti-Theft Technology\",\n    \"SoftwareGuardExtensions\": \"Software Guard Extensions ( SGX)\",\n    \"MemoryProtectionExtensionsVersion\": \"Memory Protection Extensions ( MPX)\",\n    \"OSGuardTechVersion\": \"OS Guard\",\n    \"DeviceProtectionTechBootGuardVersion\": \"Boot Guard\",\n    \"RunSureTechnologyVersion\": \"Run Sure Technology\",\n    \"ModeBasedExecutionControlVersion\": \"Mode-based Execute Control (MBE)\"\n  },\n  \"Performance Specifications\": {\n    \"CoreCount\": \"# of Cores\",\n    \"ThreadCount\": \"# of Threads\",\n    \"ClockSpeed\": \"Processor Base Frequency\",\n    \"Cache\": \"Cache\",\n    \"Bus\": \"Bus Speed\",\n    \"FSBParity\": \"FSB Parity\",\n    \"MaxTDP\": \"TDP\",\n    \"CoreVoltage\": \"VID Voltage Range\",\n    \"ScenarioDesignPower\": \"Scenario Design Power (SDP)\",\n    \"ClockSpeedMax\": \"Max Turbo Frequency\",\n    \"BusNumPorts\": \"# of QPI Links\",\n    \"TurboBoostTech2MaxFreq\": \"Turbo Boost Technology 2.0 Frequency\",\n    \"ConfigTDPMin\": \"Configurable TDP-down\",\n    \"BurstFrequency\": \"Burst Frequency\",\n    \"TurboBoostMaxTechMaxFreq\": \"Turbo Boost Max Technology 3.0 Frequency\",\n    \"UltraPathInterconnectLinks\": \"# of UPI Links\",\n    \"ConfigTDPMinFrequency\": \"Configurable TDP-down Frequency\",\n    \"FSBsSupported\": \"Supported FSBs\",\n    \"ConfigTDPMaxFrequency\": \"Configurable TDP-up Frequency\",\n    \"ConfigTDPMax\": \"Configurable TDP-up\"\n  },\n  \"I/O Specifications\": {\n    \"NumUSBPorts\": \"# of USB Ports\",\n    \"USBRevision\": \"USB Revision\",\n    \"NumSATAPorts\": \"Total # of SATA Ports\",\n    \"IntegratedLAN\": \"Integrated LAN\",\n    \"IntegratedWifi\": \"Integrated Wireless\",\n    \"GeneralPurposeIO\": \"General Purpose IO\",\n    \"UART\": \"UART\",\n    \"SATA6PortCount\": \"Max # of SATA 6.0 Gb/s Ports\",\n    \"IntegratedIDE\": \"Integrated IDE\",\n    \"USBConfigurationDescription\": \"USB Configuration\"\n  },\n  \"Networking Specifications\": {\n    \"NetworkInterfaces\": \"Interfaces Supported\",\n    \"BasebandFunctions\": \"Baseband Functions\",\n    \"RadioFreqTransceiver\": \"RF Transceiver\",\n    \"RadioFreqTransceiverFunctions\": \"RF Transceiver Functions\",\n    \"ProtocolStack\": \"Protocol Stack\"\n  },\n  \"Discrete Graphics\": {\n    \"DiscreteGraphicsFreqMhz\": \"Graphics Base Frequency\",\n    \"DiscreteGraphicsModelId\": \"Graphics Name\",\n    \"DiscreteGraphicsMaxFreqMhz\": \"Graphics Max Dynamic Frequency\",\n    \"DiscreteGraphicsComputeUnitCount\": \"Compute Units\",\n    \"DiscreteGraphicsDedicatedMemoryBandwidth\": \"Graphics Memory Bandwidth\",\n    \"DiscreteGraphicsDedicatedMemoryInterface\": \"Graphics Memory Interface\",\n    \"DiscreteGraphicsOutputOptions\": \"Graphics Output\",\n    \"DiscreteGraphics4KSupportLevel\": \"4K Support\",\n    \"DiscreteGraphicsMaxResolutionHDMI\": \"Max Resolution (HDMI)\",\n    \"DiscreteGraphicsMaxResolutionDP\": \"Max Resolution (DP)\",\n    \"DiscreteGraphicsMaxResolutionIFP\": \"Max Resolution (eDP - Integrated Flat Panel)\",\n    \"DiscreteGraphicsDirectXSupport\": \"DirectX* Support\",\n    \"DiscreteGraphicsVulkanSupport\": \"Vulkan* Support\",\n    \"DiscreteGraphicsOpenGLSupport\": \"OpenGL* Support\",\n    \"DiscreteGraphicsH264Hardware\": \"H.264 Hardware Encode/Decode\",\n    \"DiscreteGraphicsH265HardwareDesc\": \"H.265 (HEVC) Hardware Encode/Decode\",\n    \"DiscreteNumDisplaysSupported\": \"# of Displays Supported\"\n  }\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraspi%2Fscrapy-intel-ark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraspi%2Fscrapy-intel-ark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraspi%2Fscrapy-intel-ark/lists"}