https://github.com/nh2/sourceprocesswithstreams-nonthreaded-problem
Haskell: Demonstrates issue that `sourceProcessWithStreams` from conduit-extra-1.1.16 doesn't work without -threaded runtime
https://github.com/nh2/sourceprocesswithstreams-nonthreaded-problem
Last synced: 3 months ago
JSON representation
Haskell: Demonstrates issue that `sourceProcessWithStreams` from conduit-extra-1.1.16 doesn't work without -threaded runtime
- Host: GitHub
- URL: https://github.com/nh2/sourceprocesswithstreams-nonthreaded-problem
- Owner: nh2
- Created: 2017-06-26T17:40:56.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2017-06-28T15:25:54.000Z (almost 8 years ago)
- Last Synced: 2025-01-13T00:26:22.907Z (5 months ago)
- Language: Haskell
- Size: 7.81 KB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Repro: `sourceProcessWithStreams` hangs without `-threaded` runtime
Demonstrates issue that [`sourceProcessWithStreams`](https://www.stackage.org/haddock/lts-8.20/conduit-extra-1.1.16/src/Data.Conduit.Process.html#sourceProcessWithStreams) from `conduit-extra-1.1.16` doesn't work without `-threaded` runtime.
To reproduce, run `./test.sh`.
## Explanation
I found a problem with `sourceProcessWithStreams` from `conduit-extra`.
It hangs without `-threaded`. (I'm running without `-threaded` to check for some potential bug when OS threads are involved.)
I think I know what the problem is:
The [`streamingProcess`](https://github.com/fpco/streaming-commons/blob/276be069fc130f2457a667ade56343ea8e9492ac/Data/Streaming/Process.hs#L166) that it calls from `streaming-commons` does:
```haskell
_ <- forkIOWithUnmask $ \_unmask -> try (waitForProcess ph)
>>= atomically
. putTMVar ec
. either
(throw :: SomeException -> a)
id
-- return afterwards here
```
So it returns only after calling `waitForProcess`.
In the non-threaded runtime, `waitForProcess` (syscall `wait()`) means Haskell stops running.
There is also a race involved: Sometimes it works.
That happens when the code that does the actual pipe data shuffling (in `sourceProcessWithStreams`) has enough time to finish between the `forkIOWithUnmask` and the actual `waitForProcess` invocation (thus the `sleep 1` in my repro to make it always happen).
And of course it happens only if the output produced by the spawned program is larger than the OS pipe buffer (64K on Linux). Otherwise the program writes into the pipe buffer, exits, `wait()` returns and the bug does not appear.
The `System.Process` docs [even mention this](https://hackage.haskell.org/package/process-1.6.0.0/docs/System-Process.html#v:waitForProcess):
> GHC Note: in order to call waitForProcess without blocking all the other threads in the system, you must compile the program with -threaded.