I tried to create CSV upload, data processing, download function with Django

Thing you want to do

Import multiple CSVs, process them, and then return the results as CSV. This time we will implement this using the following case.

Case

The following CSV is available at a university. ʻAvailable.csvhas information on how much capacity each class has. reserved.csv` has information on how many students are planning to take classes in each class. This time, if you upload these files on the browser, you will create an application that downloads a CSV that shows how much space each class currently has.

design

On the front side, there are mainly the following two movements. ①: File upload (2): Based on the uploaded file, spit out CSV indicating the freeness of each class.

image.png

Premise

It's assumed that you already have a Django project folder (myproject) and an app folder (ʻapp`). If not, see here.

Folder structure

This time, the folder structure is as follows.

├─app
│  │  admin.py
│  │  apps.py
│  │  forms.py
│  │  functions.py
│  │  models.py
│  │  tests.py
│  │  views.py
│  │  __init__.py
│  │
│  ├─migrations
│  │    __init__.py
│  │
│  ├─static
│  │  └─upload
│  │          test.txt
│  │
│  └─templates
│       index.html
│
├─myproject
│  │  settings.py
│  │  urls.py
│  │  wsgi.py
│  │  __init__.py
│  │
│  └─static
│        humans.txt
│
└─staticfiles

Form creation

First, define the form in forms.py as follows so that you can upload the file.

from django import forms
#Define Classroom Form
class ClassroomForm(forms.Form):    
    availablity = forms.FileField() 
    reservation = forms.FileField()

Pass the form to HTML

Next, in order to display the created form on the front side, pass the form to HTML with views.py as shown below.

from django.shortcuts import render  
from app.forms import ClassroomForm  

def index(request): 
    ###Index ClassroomForm.Pass to html
    classroom = ClassroomForm()  
    return render(request,"index.html",{'form':classroom}) 

Now use {{form.as_p}} in ʻindex.html` and pull the form as follows:

<body>  
    <form method="POST" class="post-form" enctype="multipart/form-data">  
            {% csrf_token %}  
            {{ form.as_p }}  
            <button type="submit" class="save btn btn-default">Save</button>  
    </form>  
</body> 

POST processing

If you submit the form as it is, the POSTed information will be passed to the ʻindex method of views.py. So, rewrite ʻindex as follows so that POST processing is performed.

from django.shortcuts import render  
from django.http import HttpResponse  
from app.functions import process_files
from app.functions import write_into_csv
from app.forms import ClassroomForm  
from django.template import loader
import csv

def index(request):  
    if request.method == 'POST':  
        classroom = ClassroomForm(request.POST, request.FILES)
        #If there is data in the classroom
        if classroom.is_valid():  
            availability = request.FILES['availablity']
            reservation = request.FILES['reservation']
            #implement process_files
            csv_data = process_files(availability, reservation)
            #download result as a csv format             
            response = write_into_csv(csv_data)

            return response
    else:       
        ###Index ClassroomForm.Pass to html
        classroom = ClassroomForm()  
        
        return render(request,"index.html",{'form':classroom})  

Here, we are calling two methods, process_files and write_into_csv, defined in functions.py. The following defines this method.

Define logic

Define the logic used in views.py in functions.py.

import csv
from django.http import HttpResponse  

### Process csv files
def process_files(availability, reservation):  
    
        """ Description
        :type availability:
        :param availability:
    
        :type reservation:
        :param reservation:
    
        :raises:
    
        :rtype:
        """
        availability_dict = convert_to_dict(availability)
        reservation_dict = convert_to_dict(reservation)
    
        csv_data = []

        courses = list(availability_dict)
    
        for course in courses:
            remaining = availability_dict[course] - reservation_dict[course]
            row = [course, remaining]
            csv_data.append(row)
        return csv_data

def convert_to_dict(file):
    
        """ Description
        :type file:
        :param file:
    
        :raises:
    
        :rtype:
        """
        data = file.read().decode("utf-8-sig")
        lines = data.split("\r\n")
        dict = {}
        for line in lines:	
            fields = line.split(",")
            course = fields[0]
            capacity = int(fields[1])
            dict[course] = capacity

        return dict


def write_into_csv(csv_data):
        """ Description
        :type csv_data:
        :param csv_data:
    
        :raises:
    
        :rtype:
        """
        response = HttpResponse(content_type='text/csv')
        response['Content-Disposition'] = 'attachment; filename="download.csv"'
        writer = csv.writer(response)  

        for row in csv_data:
            writer.writerow(row)

        return response

With the above, the function as designed was implemented.

Demo File_Upload_Demo.gif

Finally

Please do Like if you find it helpful.

Github https://github.com/norifumi92/csv_uploader/tree/develop

reference

https://www.javatpoint.com/django-file-upload https://codepen.io/adamlaki/pen/VYpewx https://www.pythoncircle.com/post/30/how-to-upload-and-process-the-csv-file-in-django/ https://into-the-program.com/customize-input-type-file/ https://docs.djangoproject.com/en/3.0/topics/forms/

Recommended Posts

I tried to create CSV upload, data processing, download function with Django
I tried to create a table only with Django
How to create sample CSV data with hypothesis
[Data science basics] I tried saving from csv to mysql with python
I tried to save the data with discord
I tried to get CloudWatch data with Python
I tried to learn the sin function with chainer
I tried to touch the CSV file with Python
I tried to analyze J League data with Python
I tried scraping food recall information with Python to create a pandas data frame
I tried to make various "dummy data" with Python faker
I tried to automatically create a report with Markov chain
I tried to create an article in Wiki.js with SQLAlchemy
I tried to create Quip API
I tried to create Bulls and Cows with a shell program
I tried to make an image similarity function with Python + OpenCV
I tried to create serverless batch processing for the first time with DynamoDB and Step Functions
I tried to create a Power BI report by processing CSV / JSON logs using Azure Databricks
I tried to easily create a high-precision 3D image with one photo [2]. (Try processing depth with numpy)
I tried to implement Autoencoder with TensorFlow
I tried to create a linebot (preparation)
I tried to visualize AutoEncoder with TensorFlow
I tried to get started with Hy
I tried factor analysis with Titanic data!
I tried to implement CVAE with PyTorch
I tried to solve TSP with QAOA
I tried natural language processing with transformers.
I tried to create a program to convert hexadecimal numbers to decimal numbers with python
I tried to create a plug-in with HULFT IoT Edge Streaming [Development] (2/3)
I tried to create a plug-in with HULFT IoT Edge Streaming [Execution] (3/3)
I tried to send a registration completion email from Gmail with django.
I tried fMRI data analysis with python (Introduction to brain information decoding)
[Outlook] I tried to automatically create a daily report email with Python
I tried to create a plug-in with HULFT IoT Edge Streaming [Setup] (1/3)
I tried to predict next year with AI
I tried to detect Mario with pytorch + yolov3
I tried to implement reading Dataset with PyTorch
I tried to use lightGBM, xgboost with Boruta
Try using django-import-export to add csv data to django
I tried to extract named entities with the natural language processing library GiNZA
I tried to detect motion quickly with OpenCV
I tried to integrate with Keras in TFv1.1
I want to knock 100 data sciences with Colaboratory
When I tried to create a virtual environment with Python, it didn't work
I tried principal component analysis with Titanic data!
I tried function synthesis and curry with python
I tried to output LLVM IR with Python
I tried to easily create a fully automatic attendance system with Selenium + Python
I tried to detect an object with M2Det!
I tried to automate sushi making with python
I tried to predict Titanic survival with PyCaret
Write CSV data to AWS-S3 with AWS-Lambda + Python
I tried to create a button for Slack with Raspberry Pi + Tact Switch
I tried to operate Linux with Discord Bot
I tried to create an environment to check regularly using Selenium with AWS Fargate
I tried to study DP with Fibonacci sequence
I tried to start Jupyter with Amazon lightsail
I tried Django
I tried to judge Tsundere with Naive Bayes
I tried DBM with Pylearn 2 using artificial data
I tried simple image processing with Google Colaboratory.