{"id":15594958,"url":"https://github.com/gabraganca/research_tricks","last_synced_at":"2025-04-19T18:04:59.561Z","repository":{"id":68350011,"uuid":"10747991","full_name":"gabraganca/research_tricks","owner":"gabraganca","description":"Doing research like a boss.","archived":false,"fork":false,"pushed_at":"2014-03-29T20:11:20.000Z","size":300,"stargazers_count":5,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-04T10:56:57.290Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://dppg-on.github.io/research_tricks","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gabraganca.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-06-17T21:39:35.000Z","updated_at":"2020-04-29T18:45:54.000Z","dependencies_parsed_at":"2023-02-21T03:46:05.603Z","dependency_job_id":null,"html_url":"https://github.com/gabraganca/research_tricks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabraganca%2Fresearch_tricks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabraganca%2Fresearch_tricks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabraganca%2Fresearch_tricks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabraganca%2Fresearch_tricks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gabraganca","download_url":"https://codeload.github.com/gabraganca/research_tricks/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246174609,"owners_count":20735417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T00:42:22.466Z","updated_at":"2025-03-29T11:16:03.913Z","avatar_url":"https://github.com/gabraganca.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"Scientific Research Tricks\n==========================\n\nResearch is awesome! But today, due to several reasons, it is lacking some\nthings and making some error during its development.\n\nMy objective with this project is to gather some tips and tools to improve how\nresearch is done.\n\n\nCopyright 2014 DPPG@ON Organization.\n\nLicense: Creative Commons Attribution 4.0 International License.\n\nhttp://creativecommons.org/licenses/by/4.0/\n\n\nVersion Control System\n----------------------\n\nAt some point of your research did you make a bad choice or a have done mistake\nand had to go back some steps? Of course you did. Research is not done\nfollowing a recipe on some cookbook. Mistakes are done. One should try to avoid\nmistakes, but do not regret about it. Learn from it.\n\nThe problem that I want to address here is what can you do when do you do a\nmistake. You will probably want to take some steps back. If you are a very\nmethodical person, you will have every step taken wrote down on some logbook or\nsimilar. But I guess that not everybody is like this.\n\nOr maybe you keep track of changes like this:\n\n![A Story Told in file names](http://www.phdcomics.com/comics/archive/phd052810s.gif \"PhD comics #1323\")\n\nThis is not an optimized way of tracking your files. I can see only one\npossible future to this and it is a totally mess.\n\nSo, why do you not try a\n[Version Control System (VCS)](http://en.wikipedia.org/wiki/Revision_control)?\n\nA VCS [\"is the management of changes to documents, computer programs, large\nweb sites, and other\ncollections of information.\"](http://en.wikipedia.org/wiki/Revision_control).\nWith this you can keep track of any change you do on your research and fall-back\n to any point that you want.\n\nOne VCS widely used today is [`git`](http://git-scm.com/). It was created by\n[Linus Torvalds](http://en.wikipedia.org/wiki/Linus_Torvalds), the same creator\nof the [Linux Kernel](http://en.wikipedia.org/wiki/Linux_kernel). Fun fact:\nlike Linux, Linus named `git` after himself. Git is British English slang for a\nstupid or unpleasant person, and Linus said [\"I'm an egotistical bastard, and\nI name all my projects after myself. First 'Linux', now 'git'\n.\"](https://git.wiki.kernel.org/index.php/GitFaq#Why_the_.27Git.27_name.3F).\nGit is free and open source.\n\nYou can try the basics of `git`\n[here](http://try.github.io/levels/1/challenges/1) and this\n[book](http://git-scm.com/book) by [Scott Chacon](http://scottchacon.com) or\nthis\n[another book](http://chimera.labs.oreilly.com/books/1230000000561/index.html)\nby [Richard E. Silverman](http://www.qoxp.net/) can also help.\n\nBackup and Sharing\n------------------\n\nSo do you want to do a backup of your research and/or work on multiple\ncomputers? You could try a USB stick or a portable HD. That is a way to do it,\nbut maybe not the best. Maybe a cloud storage service like\n[Dropbox](https://www.dropbox.com/) or\n[Google Drive](https://drive.google.com/). I would guess that this a better way\nto do it. Dropbox has a rudimentary VCS built-in. Maybe you want to share with\nyour collaborators? You could create a shared folder on Dropbox.\n\nBut may I suggest something better? Something stronger?\n[Github](https://github.com/about).\nIt [\"is a web-based hosting service for software development projects that use\nthe Git revision control system.\"](https://en.wikipedia.org/wiki/GitHub) You\ncan store all your Git projects on Github and share with you collaborators.\nYou can browse your and others research online on your preferred browser.\nToday you almost not need to know `git` to use it.\n\nRegular expressions\n-------------------\n\nHow to find some pattern on a text.\n\n\nUseful links:\n\n* [Debbuggex](http://www.debuggex.com/): nice resource to check regex;\n* [Regex Crossword](http://regexcrossword.com/): practice or learn some regex\n  skills by playing crossword.\n\n\nShell\n-----\n\n[\"A Unix shell is a command-line interpreter or shell that provides a \ntraditional user interface for the Unix operating system and for \nUnix-like systems.\"][shell]. Some people are afraid of the \nshell. Do not. The shell is one of your most powerful ally. If you know it \nwell and dominate it, what you can do is almost magic.\n\nIf your are running Linux, you will probably have [Bash][bash] as your \n[Unix shell][shell]. If you are running Windows you can try [Cygwin][cygwin].\n\nI highly recommend seeing the [Software Carpentry][swcarpentry] tutorial \n[track on the Shell][shell_track]. It is very good and it will give you all the basics \non the Shell.\n\nFor example, below I listed some commands that can speed up your workflow.\n\n* List files on one folder and send it to a file:\n  `ls path/to/folder \u003e list_of_files.txt`;\n* Finding files:\n  `find path/to/folder -name filename`\n* Finding files with RegEx:\n  `find path/to/folder | grep 'add-your-regex-here'`\n* Finding patterns inside files with RegEx:\n  `find path/to/folder | xargs grep 'add-your-regex-here'`\n* Removing files found with previous command:\n  `find path/to/folder | grep 'add-your-regex-here' | xargs rm`\n* Compare directories:\n  `diff -rq dirA dirB`\n* Convert all figures in a folder from one type to other:\n  `for f in *.jpg; do convert ./\"$f\" ./\"${f%.jpg}.png\"; done`\n\n\nBonus Tip: [Bash git prompt](https://github.com/magicmonty/bash-git-prompt)\nis a resource that gives basic `git` information of the repository directly\non the prompt.\n\n[bash]: https://en.wikipedia.org/wiki/Bash_(Unix_shell)\n[shell]: http://en.wikipedia.org/wiki/Unix_shell\n[cygwin]: http://www.cygwin.com/\n[swcarpentry]: http://software-carpentry.org/index.html\n[shell_track]: http://software-carpentry.org/v4/shell/index.html\n\n\n\nScript programming\n------------------\n\nYou can learn python basics at\n[CodeAcademy](http://www.codecademy.com/tracks/python). I have tried and it is\nvery nice.\n\nAnother interactive way to learn python is the\n[learnpython.org](http://www.learnpython.org/). Give it a try.\n\nThere is a tutorial for non-programmers\n[here](http://en.wikibooks.org/wiki/Non-Programmer%27s_Tutorial_for_Python_3).\n\nThere is also the now famous\n[Learn Python the Hard Way](http://learnpythonthehardway.org/). It seems that\nit focus on exercise and repetition as learning tools.\n\nThis [link](http://pythonbooks.revolunet.com/) and\n[this](http://readwrite.com/2011/03/25/python-is-an-increasingly-popu#awesm=~o98NZtqHzwYofe)\ncontains a compilation of free books.\n\nIf you want to know which python modules/packages are installed in your system,\nyou just have to type in your shell:\n\n~~~\npip freeze\n~~~\n\nAnd of course, this assumes that you have\n[`pip`](https://pypi.python.org/pypi/pip) installed in your system.\n\n\nInstalling python packages on your system could result on some headache.\nFor example, I stayed one afternoon trying to figure it out why my package was\nnot running after I reinstalled it. The reason was that I was getting conflicts\nbetween the new and the old version. The solution was to use\n[virtualenv](https://pypi.python.org/pypi/virtualenv) and\n[`virtualenvwrapper`][venvw].\n\n[venvw]: http://virtualenvwrapper.readthedocs.org/en/latest/index.html\n\nas the name says `virtualenv` creates a python virtual environment. this allows\nthe user to install any python package without worryng in messing the system\npackages. also, the user can create any number of virtual environments,\nfor example, one for each project.\n\nGood editor\n-----------\n\nit is very important to have a good editor.\n\nif you are the nerd-geek-kind-of-awesome guy, you should probably try\nvim or emacs.\n\nAutomation\n----------\n\nhow to automate things using python. this is great when you do a miskate and\nhad to run several things again.\n\nsome links to check:\n\n* [@astrofrog course](https://github.com/astrofrog/py4sci)\n* [@jrjohansson tutorials](https://github.com/jrjohansson/scientific-python-lectures)\n\n\nnegative results\n-------------\n\nnegative results are still results. publish them. why publish it?\njust look the following cartoon and i guess you will understand.\n\n![negative data](http://upmic.files.wordpress.com/2013/06/negative-data.png \"from the upturned microscope\")\n\ngot it?\n\nprobably, a journal will not accepted a paper showing negative results. and\nbecause of this, enters [figshare](http://www.figshare.com).\nThis resource [\"allows users to upload any file format to be made visualisable\nin the browser so that figures, datasets, media, papers, posters, presentations\nand filesets can be disseminated in a way that the current scholarly publishing\nmodel does not allow.](http://figshare.com/about). Each image, presententation\nor any other kind of data receives a\n[DOI](http://en.wikipedia.org/wiki/Digital_object_identifier)\nthat will uniquely identify your data and it will allow others to cite it.\n\nVisualization\n-------------\n\nHow to visualize your data is a very important step. I must say, crucial. It is\nvery hard to obtain any results without it. Unless you are a statistic ninja,\nand even that, you would probably use visualization to convince your audience\nof your findings.\n\nOne of the fathers of this field is\n[Edward Tufte][Tufte]. I have not yet read it, but his first book, [The Visual\nDisplay of Quantitative Information][Tufte_book], is very highly recommended.\n\n[Tufte]: http://www.edwardtufte.com/tufte/index\n[Tufte_book]: http://www.edwardtufte.com/tufte/books_vdqi\n\nSo, how one can visualize data? There are several tools available. You could\nuse a spreadsheet editor, like [LibreOffice Calc][OOffice]. The resources are\nlimited, but you can get a visualization quickly and it is very good when one\ndoes not have any other skils. And if you want to make better plots with it,\nyou will probably want to try [RAW](http://raw.densitydesign.org/).\n\n[OOffice]: https://www.libreoffice.org/features/calc/\n\nOne can draw it by hand, or use [GIMP](http://www.gimp.org/) or\n[Inkscape](http://inkscape.org/), for example. But, for that, it would probably\nbe good to have some artistic skills. And also, if something change on your\ndata, you would probably have to start it from scratch.\n\nAlso, one could code some routine to do visualization. The advantage of this\nmethod is that probably you will only have to code it once, i.e. the code is\ndata independent. If the data changes, you only have to run it again. Well,\nachieving this will only depend of your coding skill.\n\nThere are several good visualization toolkits on the wild, like\n[`D3`](http://d3js.org/) for javascript. If you want to learn D3, the book by\n[Scott Murray](http://alignedleft.com/about/),\n_Interactive Data Visualization for the Web_, is now\n[available online for free][d3_free]. And, for python, we have\n[`Matplotlib`](http://matplotlib.org/index.html),\n[`Mayavi`](http://code.enthought.com/projects/mayavi/)\nand [`ggplot`](https://github.com/yhat/ggplot?source=cc#ggplot-from-yhat).\n[`Ggplot2`](http://ggplot2.org/) is a very well-known plotting system for\n[R][Rlang] and `ggplot` is specially good for those that are learning python\nbut have a solid background in R. And if you want the `D3` awesomeness in your\npython code, try [Vincent](https://github.com/wrobstory/vincent).\n\n[Rlang]: http://www.r-project.org/\n[d3_free]: http://chimera.labs.oreilly.com/books/1230000000345/index.html\n\n\nAlso, [Bret Victor](http://worrydream.com/#!/Bio) is working on a software\nthat will allow to dynamically draw your visualizations. If you want to know\nmore about it, you can check his [talk](http://vimeo.com/66085662) and\n[here](http://worrydream.com/DrawingDynamicVisualizationsTalkAddendum/).\nIt is really impressive!\n\nAnd according to [Nathan Yau](https://twitter.com/flowingdata) from\n[Flowing Data](http://flowingdata.com/), there is only one way to learn\nvisualization: [work with data][fd_post].\n\n[fd_post]: http://flowingdata.com/2013/07/12/getting-started-with-visualization-after-getting-started-with-visualization/\n\nI also recommend his [first book](http://flowingdata.com/visualize-this/) as a\nstarting point to do visualization. He address what type of visualization there\nare, the tools available and all on a very pleasant reading content.\nI have not yet read his [second](http://flowingdata.com/data-points/), but I am\nvery anxious to get a copy.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgabraganca%2Fresearch_tricks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgabraganca%2Fresearch_tricks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgabraganca%2Fresearch_tricks/lists"}