I was designing a web application where i need a simple button to run a bunch of python code resides in Unix system and to show log messages of running scripts to my web application. This can be done in many ways, but i wanted to stick on python and its supported modules.
So, last few days i spent time on learning pyjamas and to configure pyjamas with jsonrpc and mod-python. That really kept me interested as it was totally new to me and lerning pyjamas (i was thinking ot it since a year!!!) is great to me.
I have gone through all sites, tutorials and everything but didn't get any good way to do it on single page and i am sure its really difficult for new programmer to do the same(examples are just to know this can be done but not well explained).
So, i am just writing down things i did to configure pyjamas, jsonrpc and mod python.
The first thing to do is, install jsonrpc, simplejson, mod_python(configured with apache) and pyjamas.
First i design frontend of my application using pyjamas.
1. create a folder VLayouts, then paste following code to VLayouts.py
import pyjd # this is dummy in pyjsfrom pyjamas.ui.RootPanelimport RootPanel
from pyjamas.ui.SimplePanelimport SimplePanel
from pyjamas.ui.Imageimport Image
from pyjamas.ui.Labelimport Label
from pyjamas.uiimport HasAlignment
from pyjamas.ui.Buttonimport Button
from pyjamas.ui.CheckBoximport CheckBox
from pyjamas.ui.VerticalPanelimport VerticalPanel
from pyjamas.ui.HorizontalPanelimport HorizontalPanel
from pyjamas.ui.HTMLimport HTML
from pyjamas.ui.DockPanelimport DockPanel
from pyjamas.uiimport HasAlignment
from pyjamas.ui.FlowPanelimport FlowPanel
from pyjamas.ui.HTMLPanelimport HTMLPanel
from pyjamas.ui.ScrollPanelimport ScrollPanel
from pyjamas.ui.DisclosurePanelimport DisclosurePanel
from pyjamas import DOM, Window
from pyjamas.ui.TextBoximport TextBox
from pyjamas.ui.PasswordTextBoximport PasswordTextBox
from FileOpenDlg import FileOpenDlg
import Utils
from pyjamas.JSONServiceimport JSONProxy
class VLayoutsJSONProxy(JSONProxy):
def__init__(self):
## services/vlayouts.py counts from output folder
JSONProxy.__init__(self, "services/vlayouts.py", ["callMethod","test"])#JSONProxy.__init__(self, "./services/vlayouts.py", ["callMethod","test"])class VLayoutsService:
def__init__(self, callback):
self.callback = callback
self.proxy = VLayoutsJSONProxy()def test(self):
self.proxy.test(self)def callMethod(self,cmd):
Window.alert("inside VLayoutsService: callMethod")self.callback.showStatus("Calling Method!!!")
a = self.proxy.callMethod("ls -lth", self)self.callback.showStatus(a)def onRemoteResponse(self, response, request_info):
Window.alert("inside VLayoutsService: onRemoteResponse")
Window.alert(dir(request_info))
Window.alert(request_info.method)
Window.alert(request_info.handler)
Window.alert(response)if request_info.method == "callMethod":
self.callback.showStatus("Called method %s" % request_info.method)else:
self.callback.showStatus(""" REQ METHOD = %s RESP %s """ % (request_info.method,response))def onRemoteError(self, code, errobj, request_info):
Window.alert("inside VLayoutsService: onRemoteError")
Window.alert(request_info)
message = errobj['message']
Window.alert(message)if code != 0:
self.callback.showStatus("HTTP error %d: %s" % (code, message))else:
json_code = errobj['code']self.callback.showStatus("JSONRPC Error %s: %s" % (json_code, message))class VLayouts(SimplePanel):
def__init__(self):
SimplePanel.__init__(self)self.VLayoutsService = VLayoutsService(self)def onModuleLoad(self):
text="Vivek Sharma"self.status = Label()
contents = HTML(text)
scroller = ScrollPanel(contents, StyleName="ks-layouts-Scroller")
panel = DockPanel(BorderWidth=4, Padding=20,Width="100%",Height="100%",
HorizontalAlignment=HasAlignment.ALIGN_CENTER,
VerticalAlignment=HasAlignment.ALIGN_MIDDLE)
north = Image("./myImage.jpeg",Width="800px", Height="100px",)
north.addClickListener(getattr(self, "onImageClicked"))#north = Label("Image should be placed here")
west = Button("RUN FUNCTION", self.onButtonClick)
fileLocation = "logfile.log.txt"
center = FileOpenDlg(fileLocation = fileLocation)##center = Button("SHOW LOG", self.openLogFile)#east = Label("East")#south = Label("South")
panel.add(north, DockPanel.NORTH)
panel.add(west, DockPanel.WEST)
panel.add(center, DockPanel.CENTER)#panel.add(east, DockPanel.EAST)#panel.add(south, DockPanel.SOUTH)
panel.setCellHeight(north, "100px")
panel.setCellHeight(center, "400px")
panel.setCellWidth(center, "800px")
panel.setCellWidth(west, "200px")
panel.setCellVerticalAlignment(west, '100px')#HasAlignment.ALIGN_TOP)self.add(panel)#Window.alert("Hello, AJAX!")
panel2 = VerticalPanel()
panel2.add(self.status)
RootPanel().add(panel2)
RootPanel().add(panel)def showStatus(self, msg):
self.status.setText(msg)def onButtonClick(self,sender):
Window.alert("onButtonClick function called")self.VLayoutsService.callMethod("ls -lth")def openLogFile(sender):
fileLocation = "logfile.log.txt"
dlg = FileOpenDlg(fileLocation = fileLocation)
dlg.show()def onShow(self):
passdef onImageClicked(self):
Window.alert("Vivek Sharma!!!")def makeLabel(self, caption):
html = HTML(caption)
html.setStyleName("ks-layouts-Label")return html
if__name__ == '__main__':
pyjd.setup("./VLayouts.html")
app = VLayouts()
app.onModuleLoad()
pyjd.run()
configuration i did in /opt/vivek/apache/conf/httpd.conf
<Directory "/opt/vivek/apache/cgi-bin">
AddHandler default-handler .jpeg .png .css .html .py .js .txt
PythonHandler jsonrpc
AllowOverride All
Options Indexes FollowSymLinks
Order allow,deny
Allow from all
</Directory>
Now, create .htaccess in /opt/vivek/apache/cgi-bin/VLayouts/public/services folder with content
Options FollowSymLinks
PythonPath "['/opt/vivek/apache/cgi-bin/VLayouts/output/services']+sys.path"
AddHandler mod_python .py
PythonHandler vlayouts
PythonDebug On
and check the changes in /opt/vivek/apache/cgi-bin/VLayouts/output/services/.htaccess , or directly create .htaccess here only with above content
create vlayouts.py in /opt/vivek/apache/cgi-bin/VLayouts/public/services folder with following content
#! /usr/bin/env pythonimport logging
logging.basicConfig(filename="/home/vivek/Desktop/contactjson.log", level=logging.DEBUG)
logging.debug("OLDLoading contact service")from test import AB
class Service:
def callMethod(self, msg):
return"msg"if__name__ == '__main__':
# this is if JSONService.py is run as a CGIfrom jsonrpc.cgihandlerimport handleCGIRequest
handleCGIRequest(Service())else:
# this is if JSONService.py is run from mod_python:# rename .htaccess.mod_python to .htaccess to activate,# and restart Apache2from jsonrpc.apacheServiceHandlerimport handler
run as /home/vivek/Desktop/TGZS/pyjamas-0.7/bin/pyjsbuild VLayouts.py
Recently i was running a script which uses "/tmp" heavily for all file processing and rolling-up. So at each interval of time i need to know space available to my disc. Looks very simple to run "df -h" every time on console, but i was curious to write python script for this and the result is:
importosimportsysfrom time import sleep
cmd = 'df -h'
print"Press ctrl-C to exit..\n"try:
while1:
printos.system(cmd)print '\n'
sleep(100)exceptKeyboardInterrupt:
print"Good Bye"
Comments in your code are always good for new person to understand the logic and flow of program. I guess, people likes programming more, if they comment properly, same way as i do. Proper commenting will not allow you to read code line by line. The best way to comment is to comment in such a way, you can extract it easily. I do comments in my python code using '##' which makes me easy to extract them from script. Following code extracts my comment:
importosimportsys## python extract_comment.py filename comment_delimeter
filename = sys.argv[1]
sepr = sys.argv[2]
fl = open(filename,'r')
fl_con = fl.readlines()for row in fl_con:
if sepr in row:
ind = row.find(sepr)print row[ind:]
fl.close()
copy the code in file extract_comment.py, filename is the python script from where you need to extract comments and comment_delimeter is '##' in my case.
Recently, I was hanging arround flex codes which calls python script resides on other server through web services. I got confused, Is it a good idea to use web service just to call python script from other server? Why not to use cgi module or mod-python to get the same result as getting through web services?
So i decided to write a simple web server which has some methods to be called as a URL. Got excellent help from
http://fragments.turtlemeat.com/pythonwebserver.php
then, i added some code.
importstring,cgi,time
fromosimport curdir, sep
fromBaseHTTPServerimport BaseHTTPRequestHandler, HTTPServer
class VivekServer(BaseHTTPRequestHandler):
def do_GET(self):
try:
ifself.path == '/fetch':
self.send_response(200)self.send_header('Content-type', 'text/html')self.end_headers()
res = self.wcount()self.wfile.write("Number of count for 'anyword' :")self.wfile.write(res[0])self.wfile.write(" url is :")self.wfile.write(res[1])returnifself.path == '/calculate':
self.send_response(200)self.send_header('Content-type', 'text/html')self.end_headers()
res = self.calculate()for each in res:
self.wfile.write(each)self.wfile.write('\n')returnreturnexceptIOError:
self.send_error(404,'File Not Found: %s' % self.path)def calculate(self):
importrandom
WORD = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
data = []for i inrange(1, 100):
data.append((random.randrange(0, 1000), random.sample(WORD, len(WORD))[0]))return data
def do_POST(self):
passdef wcount(self):
from BeautifulSoup import BeautifulSoup as soup
import urllib2
htm = 'http://www.anyurl.com'
html_text = urllib2.urlopen( htm ) .read()
sp = soup(html_text)
idea = sp.findAll("anyword")
all_a = [ each.get('href')for each in sp.findAll('a')]
num = 0for each in all_a:
if each.find("anyword") > 0:
num=num+1return(idea,num,htm)def main():
try:
server = HTTPServer(('', 7999), VivekServer)print 'Server Started.....'
server.serve_forever()exceptKeyboardInterrupt:
print 'Server Ends.....'
server.socket.close()if__name__ == '__main__':
main()
This is just a help to start a simple web server and call your method through url.
To run above code, just do, python abovecode.py
open web browser, type url as
http://localhost:7999/fetch
http://localhost:7999/calculate
Today, while working with csv files i got into fantastic situation where i have a list of million values and i make iteration on that. So i thought, it would be easy for me if i split list into number of pieces which dont affect my code, memory and CPU. I can also use generator expression to make make my code run faster, but i was very curious to write code to split list into number of pieces(uses little of generator exp).
And the result is here -
importos,sysdef split_seq(seq, num_pieces):
""" split a list into pieces passed as param """
start = 0for i inxrange(num_pieces):
stop = start + len(seq[i::num_pieces])
yield seq[start:stop]
start = stop
seq = [i for i inrange(100)]## define your list here
num_of_pieces = 3for seq in split_seq(seq, num_of_pieces):
printlen(seq), '-> ',seq
Recently i had gone through a situation like to split a 40GB csv file
for further processing, into 60 pieces having name/content to be decided
dynamically based on a id present in that csv file. That really make me
to write some little code to open/write/close file dynamically. I wrote
following code to achieve this.
importos, sys
a=range(10)for each in a:
s = "fl_%s = open('%s','a')" % (each,each)exec s
exec"fl_%s.write('%s')" % (each,each)
com = "fl_%s.close()" % (each)exec com
# can also check if file is closed or open by# com = "fl_%s.closed" % (each)# bool(com) #return true if file is closed else false
While working with csv module of python i got very interesting thing about join.
I was reading a huge csv file line by line and for some kind of operation i converted that row to list and again that list to string. But, my row consists of some integer values so i always get
TypeError: sequence item 5: expected string, int found
So, i am writing small code to let new guys know about this.
i have a list and i want to join this.
ls=['a','b',4,'c']
','.join(ls)
ends up with : TypeError: sequence item 2: expected string, int found
Recently, i found excellent use of Twill module
of python. I have used twill before, just to check multiple login
functionality of one of my plone site. That was to check load on my
login script.
But, some days back, i used Twill to fetch multiple user detail from proxy site.
Here, i am just going to explain the little code, which i wrote. It might be helpful for others.
What i am doing here??
i am opening google.com and search the term "Twill"
Here, the first thing is, how to use twill module in python code?
download Twill from http://twill.idyll.org/
import twill
import twill.commands
t_com = twill.commands## get the default browser
t_brw = t_com.get_browser()## open the url
url = 'http://google.com'
t_brw.go(url)## get all forms from that URL
all_forms = t_brw.get_all_forms()## this returns list of form objects## now, you have to choose only that form, which is having POST methodfor each_frm in all_forms:
attr = each_frm.attrs## all attributes of formif each_frm.method == 'POST':
ctrl = each_frm.controls## return all control objects within that form (all html tags as control inside form)for ct in ctrl:
if ct.type == 'text': ## i did it as per my use, you can put your condition here
ct._value = "twill"
t_brw.clicked(each_frm,ct.attrs['name'])## clicked takes two parameter, form object and button name to be clicked.
t_brw.submit()## you might write the output (submitted page) to any file using content = t_brw.get_html()## dont forget to reset the browser and putputs.
t_com.reset_browser
t_com.reset_output
What will you do if you are creating dictionary structure dynamically, and it got millions of keys?
Accessing that dictionary later in your code might get some resource. can't it?
I also hanged on this kind of situation and my dictionay got 10K millions key. So i used dictionary as generator to make my work easy.
Folloing code just explain how to use dictionary as generator.
a=range(100000)
b=range(100000)
c=dict(zip(a,b)) #create dictionary with 100000 keys
d_len=len(c)
d_keys = (k for k in c.keys()) # generator expression
for i in range(d_len):
key = d_keys.next()
.
.
.
## do your operation on keys
I am not good at core java programming, but good at "Hello World" kind of program :) .
SO i wrote a Java program to call python script(can also pass arg values). Take a look.
This is Java code
import java.io.*;
// run this way
// javac JavaRunCommand.java
// java -classpath . JavaRunCommand
public class JavaRunCommand {
public static void main(String args[]) {
String st = null;
try {
String[]callAndArgs= {\"python\",\"my_python.py\",\"arg1\",\"arg2\"};
Process p = Runtime.getRuntime().exec(callAndArgs);
BufferedReader stdInput = new BufferedReader(new
InputStreamReader(p.getInputStream()));
BufferedReader stdError = new BufferedReader(new
InputStreamReader(p.getErrorStream()));
// read the output
while ((s = stdInput.readLine()) != null) {
System.out.println(s);
}
// read any errors
while ((s = stdError.readLine()) != null) {
System.out.println(s);
}
Renaming multiple file once is really little confusing using
command-line. There are lots of way to do it via programming, but yet, i
didnt fine any on-the-spot command to do it.
So i used python to do it simply.
My requirement was actually:
1) i have one dedicated folder, where i have to rename all files.
2) all filename to be renamed are structured, i mean, i have to rename all dedupe_<number>.csv to <number>.csv
I did this using following code,
import os
from os import listdir, getcwd, rename
list_files = listdir(getcwd())
for filename in list_files:
if not filename.startswith('.') and 'dedupe_' in filename:
ext = filename.split('.')[-1]
new_name = ''.join(filename.split('.')[:-1]).replace('dedupe_','')+'.'+ext
cmd = 'mv '+filename + ' ' +new_name
os.popen(cmd)
If you can read this post, it means that the registration process was successful and that you can start blogging
Who am I?
I am a python programmer, my experience ranges from developing plone product to data/text analysis.I also worked on data crunching, python to database interaction(MySQL, Oracle), CGI, Flask, crawler and all basic python packages.