Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/krolow/marsvin
Structural Crawler framework written in PHP
https://github.com/krolow/marsvin
crawler framework parser php
Last synced: about 1 month ago
JSON representation
Structural Crawler framework written in PHP
- Host: GitHub
- URL: https://github.com/krolow/marsvin
- Owner: krolow
- Created: 2012-09-17T18:20:02.000Z (over 12 years ago)
- Default Branch: master
- Last Pushed: 2013-08-05T03:21:26.000Z (over 11 years ago)
- Last Synced: 2024-04-08T16:10:22.202Z (9 months ago)
- Topics: crawler, framework, parser, php
- Language: PHP
- Homepage:
- Size: 174 KB
- Stars: 12
- Watchers: 2
- Forks: 5
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
#Marsvin
## What is it?
Have you ever write a crawler or parser?
If yes, you must know that is always a trivial task, but we have always to think how structure our code to do such a thing...
So... to solve that Marvins was created, Marvins provide a simple API and structure to be followed to you create your parsers or crawler. The main focus is to facilitate the task of parser data from external resources, to extract data from websites or import data from XML, CSV files etc...
And it has more, as a plus it comes with a Process management, that makes enable you open more than one PHP process, so you are able to do more than one thing at the same time.
## How to use it?
**Create a composer.json**
```javascript
{
"name" : "your/projectname",
"require" : {
"cobaia/marsvin" : "dev-master"
},
"minimum-stability" : "dev"
}
```
**Run the command:**
```bash
composer.phar install
```
**Create your console command:**
**File:** console.php
```php
addCommands(
array(
new Marsvin\Command\GenerateProviderCommand(),
new Marsvin\Command\RequestProviderCommand(),
)
);
//You are able to pass as much helper set you want, like Doctrine, Monolog, etc...
//$console->setHelperSet($helperSet);
$console->run();
```
After create the console you are already enable to run the command: php app/console.php
You will check that we have two commands to use:
```bash
marsvin
marsvin:generate:provider Generate Provider code structure
marsvin:request:provider Request one specific Provider
```
**Marsvin use the following nomenclature:**
- **Provider:** It's the name of the operation that you will be doing, an example of provider would be: Facebook, Github, Google, etc..
- **Requester:** It's the layer responsible to make the requests of one provider, for example for Facebook would be one HTTP Request, for Github maybe it can be one GIT operation, for another provider it can be one FTP access etc...
- **Parser:** Once you the request operation has been done, the parser layers comes this layer will take care of the data, so if you want to setup some entities of doctrine, or do you want to create some array of datas, etc... Or you want to normalize some how the data, this is the layer that you will be using to do such a task.
- **Persister:** Once you have parsed your data, it goes to the Persister layer, here is where you will do forever you want with the data, in doctrine for example you will be able to persist and flush the data into database, or if you want to persists in file system, persists sending one email, or whatever the Persister layer will be handling that to you.
### Creating our first provider
To create our provider, marvins has one command that create the folder structure to you:
```bash
php app/console marsvin:generate:provider MyProject\\Github ./src/
```
You will check that Marsvin will generate the following folder tree to you:
```bash
.
└── MyProject
└── Github
├── GithubParser.php
├── GithubPersister.php
├── GithubProvider.php
└── GithubRequester.php
```
Now it's time to setup the adapters, Marsvin using the adapter pattern to define each one of the layers, so for example if you want to make HTTP Request you can setup one HttpAdapter to use in the requester layer.
By default Marvins comes with few adapters:
- **Requester:** DefaultAdapter.php BuzzAdapter.php
- **Parser:** DomAdapter.php
- **Persister:** DefaultAdapter.php DoctrineAdapter.php
So let's setup our provider:
```php
getAdapter();
$profiles = array(
'krolow',
'gquental',
'moacirosa',
'fabpot',
);
$self = $this;
foreach ($profiles as $profile) {
$this->process(function () use ($self, $adapter, $profile) {
$self->done(
new Response(
$adapter->request(
sprintf(
GithubRequester::GITHUB_URL,
$profile
)
)
)
);
});
}
}
}
```
The Parser:
```php
getAdapter();
$dom = $adapter->parse($response->get());
$xpath = new DOMXPath($dom);
$nodes = $xpath->query('//span[@itemprop="name"]');
$author = $nodes->item(0)->nodeValue;
$nodes = $xpath->query('//li[contains(@class, "public")]');
$projects = array();
foreach ($nodes as $node) {
array_push(
$projects,
$this->parseProject($node, $author)
);
}
$this->done(new Response($projects));
}
protected function parseProject($node, $author)
{
$doc = new DOMDocument();
$doc->appendChild($doc->importNode($node, true));
$name = $doc->getElementsByTagName('h3');
$url = $doc->getElementsByTagName('a');
$project = array(
'name' => trim($name->item(0)->nodeValue),
'url' => 'http://github.com/' . $url->item(2)->getAttribute('href'),
'author' => $author
);
return $project;
}
}
```
The Persister:
```php
getAdapter();
$adapter->persist($response->get());
file_put_contents('/tmp/marsvin.log', var_export($adapter->flush(), true), FILE_APPEND);
}
}
```
To run the command do the follow:
```bash
php app/console.php marsvin:request:provider MyProject\\Github\\GithubProvider
```
You can check what is happening here:
```bash
tail -f /tmp/marsvin.log
```