shellpy

A tool for convenient shell scripting in Python. It allows you to write all your shell scripts in Python in a convenient way and in many cases replace Bash/Sh.

Preface - Why do we need shell python?

For many people bash/sh seem to be pretty complicated. An example would be regular expressions, working with json/yaml/xml, named arguments parsing and so on. There are many things that are much easier in python to understand and work with.

Introduction

Shell python has no differences from python except for one. Grave accent symbol (`) does not mean eval, it means execution of shell commands. So

`ls -l`

will execute ls -l in shell. You can also skip one ` in the end of line

`ls -l

and it will also be a correct syntax. It is also possible to write multiline expressions

`
echo test > test.txt
cat test.txt
`

and long lines

`echo This is \
  a very long \
  line

Every shellpy expression returns a Result

result = `ls -l

or normally raises an error in case of non zero output of a command

try:
  result = `ls -l non_existent_file
except NonZeroReturnCodeError as e:
  result = e.result

The result can be either Result or InteractiveResult. Let's start with a simple Result. You can check returncode of a command

result = `ls -l
print result.returncode

You can also get text from stdout or stderr

result = `ls -l
result_text = result.stdout
result_error = result.stderr

You can iterate over lines of result stdout

result = `ls -l
for line in result:
    print line.upper()

and so on.

Integration with python and code reuse

As it was said before shellpython does not differ a lot from ordinary python. You can import python modules and use them as usual

import os.path

`mkdir /tmp/mydir
os.path.exists('/tmp/mydir') # True

And you can do the same with shellpython modules. Suppose you have shellpy module common as in examples directory. So this is how it looks

ls common/
common.spy  __init__.spy

So you have directory common and two files inside: __init__.spy and common.spy. Looks like a python module right? Exactly. The only difference is file extension. For __init__.spy and other files it must be .spy. Let's look inside common.spy

def common_func():
    return `echo 5

A simple function that returns Result of echo 5 execution. How is it used how in code? As same as in python

from common.common import common_func

print('Result of imported function is ' + str(common_func()))

Note that the common directory must be in pythonpath to be imported.

How does import work?

It uses import hooks described in PEP 0302 -- New Import Hooks. So, whenever importer finds a shellpy module or a file with .spy extension and with the name that you import, it will try to first preprocess it from shellpy to python and then import it using standard python import. Once preprocessed, the file is cached in your system temp directory and the second time it will be just imported directly.

Important note about import

Import of shellpython modules requires import hook to be installed. There are two way how to do it: - run shellpython scripts with the shellpy tool as described below in the section Running - run your python scripts as usual with python but initialize shellpython before importing any module with shellpython.init() as in the Example

Example

This script clones shellpython to temporary directory and finds the commit hash where README was created

```python

import tempfile import os.path from shellpython.helpers import Dir

We will make everything in temp directory. Dir helper allows you to change current directory

withing 'with' block

with Dir(tempfile.gettempdir()): if not os.path.exists('shellpy'): # just executes shell command `git clone https://github.com/lamerman/shellpy.git

# switch to newly created tempdirectory/shellpy
with Dir('shellpy'):
    # here we capture result of shell execution. log here is an instance of Result class
    log = `git log --pretty=oneline --grep='Create'

    # shellpy allows you to iterate over lines in stdout with this syntactic sugar
    for line in log:
        if line.find('README.md'):
            hashcode = log.stdout.split(' ')[0]
            print hashcode
            exit(0)

    print 'The commit where the readme was created was not found'

exit(1) ```

Two lines here are executed in shell git clone https://github.com/lamerman/shellpy.git and git log --pretty=oneline --grep='Create'. The result of the second line is assigned to variable log and then we iterate over the result line by line in the for cycle

Installation

You can install it either with pip install shellpy or by cloning this repository and execution of setup.py install. After that you will have shellpy command installed.

Running

You can try shellpython by running examples after installation. Download this repository and run the following command in the root folder of the cloned repository:

shellpy example/curl.spy

shellpy example/git.spy

There is also so called allinone example which you can have a look at and execute like this:

shellpy example/allinone/test.spy

It is called all in one because it demonstrates all features available in shellpy. If you have python3 run instead:

shellpy example/allinone/test3.spy

Documentation

Wiki

Compatibility

It works on Linux and Mac for both Python 2.x and 3.x. It should also work on Windows.

Issues

Ability to set python executable that will run processed scripts

opened on 2016-03-06 15:44:52 by lamerman

Now it is hardcoded in header_root.tpl with pretty generic #!/usr/bin/env python . But maybe it should be possible to configure it.

How to make automatic testing of shellpy functions

opened on 2016-03-03 21:58:12 by lamerman

We need to think how automatic testing of shellpy functions will be done and create a couple of examples of tests. Also some small article on it is needed in wiki.

Stacktrace in case of error should possibly not include shellpy internals

opened on 2016-03-02 21:06:37 by lamerman

Here is an example

^CTraceback (most recent call last): File "/tmp/shellpy_root/root/cr/scripts/monitor.py", line 272, in Traceback (most recent call last): File "/usr/bin/shellpy", line 9, in load_entry_point('shellpy==0.4.5', 'console_scripts', 'shellpy')() File "/usr/lib/python2.7/site-packages/shellpython/shellpy.py", line 40, in main main() retcode = subprocess.call(processed_file + ' ' + ' '.join(script_args), shell=True, env=new_env) File "/usr/lib64/python2.7/subprocess.py", line 524, in call File "/tmp/shellpy_root/root/cr/scripts/monitor.py", line 269, in main watchgod() File "/tmp/shellpy_root/root/cr/scripts/monitor.py", line 232, in watchgod return Popen(_popenargs, _kwargs).wait() File "/usr/lib64/python2.7/subprocess.py", line 1376, in wait sleep(30) KeyboardInterrupt pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0) File "/usr/lib64/python2.7/subprocess.py", line 478, in _eintr_retry_call return func(args) KeyboardInterrupt

Possibly it should start with the line File "/tmp/shellpy_root/root/cr/scripts/monitor.py", line 269, in main

Interference between shell and python syntax

opened on 2016-03-01 17:53:42 by gpalsingh

using: echo {1..9} causes an exception to be thrown because python interprets it as a format specifier.

Think about races with preprocessed files

opened on 2016-02-17 22:29:03 by lamerman

What if two shellpy processes will write one preprocessed file to cache in one time?

Stdout should be immediately available even if in non interactive mode

opened on 2016-02-17 21:16:07 by lamerman

Now stdout in non interactive mode is not printed immediately, it waits when the process finishes. It would be great to see stdout, stderr as soon as it comes.